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Abstract 


The Teacher Leaders Corps had many strengths and was fairly successful in reaching its implementation 
and short-term goals. Central training quality was considered strong and attendance rates were high 
(especially in Years 1 and 2). Of those who began the TLC work in 2013-14, approximately 60% 
participated all three years. At the school level, about half of the intended dissemination events took 
place. Professional learning sessions emphasized use of various digital resources. Discovery Education 
resources were available to all schools and provided evidence that online resources were utilized by 
teachers and students. DE utilization was higher in Year 1 than in Year 2 or 3; and teachers utilized DE 


more than students. Decreased use of DE after Year 1 could reflect less interest or an increase in digital 
resources available to schools. TLC members utilized technology appropriately and in a variety of ways 
based on classroom observations. The school team model utilized in TLC provided a better opportunity 
for sustaining the effort than training sessions with no follow-up. Ways to further increase the 
likelihood of implementation, impact, and sustainability in schools include putting structures in place at 
the central and school level to facilitate and monitor teacher implementation and to provide more 
coaching for teachers. 
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Program Description 


The Teacher Leader Corps (TLC) professional learning initiative focused on building teachers’ skills in the 
appropriate integration of technology into instruction. WCPSS Instructional Technology and Media 
Services (ITMS) staff contracted with Discovery Education (DE) to serve as trainers of four-member TLC 
teacher teams from each school. While some professional learning opportunities utilized DE materials, 
training also covered other technology resources. Costs for the three-year initiative exceeded one 
million dollars per year. DE trainers provided professional learning opportunities through multi-day 
training for each of the three years to the TLC members. TLC members subsequently shared TLC ideas 
and techniques through with other staff in their schools. 


The primary goal of TLC was to build a sense of efficacy among teachers in knowing how to 
appropriately utilize technology for instruction (see Figure 1). The expectation was that teachers would 
then increase meaningful use of digital resources in instruction, which would lead to increased student 
engagement in their learning, and ultimately, increased student learning and achievement. 


Figure 1 
Pathway of Change 
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Multiple studies suggest that the adoption of technology and digital media provide many advantages to 
teachers, students, and school districts when implemented in a proper and sustainable manner. 
Instructional technology use gives students access to the tools that they will be expected to utilize 
effectively after graduation as well as increasing student achievement (Campuzano, Dynarski, Agodini, & 
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Rall, 2009; Shapley, Sheehan, Maloney, & Camikas-Walker, 2010; Suhr, Hernandez, Grimes, & 
Warschauer, 2010; Warschauer, 2006; Weston & Bain, 2010). 


Implementation 


Implementation of TLC sessions for teacher teams at the central level was strong. Planned full-day TLC 
trainings occurred for multiple cohorts each year. Attendance was 66% to 71% in Years 1 and 2, with 
some decline in Year 3 (47%). Participants had positive views of the sessions and considered them 
helpful. Of the original 695 Year 1 TLC members, 527 continued participation in Year 2, and 407 
participated in Year 3 to complete their three-year commitment. Thus, over half (58.6%) stayed with TLC 
all three years. Replacement team members were added in most but not all cases. In some cases, TLC 
teams included fewer than the expected four members for an extended amount of time (i.e. several 
months to a whole school year). 


At the school level, the expectation was that TLC members would share training modules and provide 
demonstrations in learning labs for their school colleagues. Completion was moderate with 
approximately 60% of the expected offerings made in Year 1 and a decline in Years 2 and 3 to 35% and 
43% respectively. Time was the primary reason teachers gave for not disseminating the learning—most 
commonly time to share at staff meetings and/or time for other teachers to visit the learning labs. Some 
teachers mentioned that the school administration did not consider the TLC work as high in priority as 
other initiatives. Finally, some teachers also noted that they did not have enough time to prepare to 
share the material. 


Teacher Outcomes 


Teachers’ sense of self-efficacy in utilizing digital resources for instruction increased for both the TLC 
teachers and non-TLC teachers. The TLC and non-TLC groups started out at about the same point in 
terms of their self-evaluations, and both groups improved over time. TLC members reached a higher 
level after the first year of TLC training and maintained this higher level across the three years. Teachers 
who were only members of TLC for one or two years had similar patterns of self-efficacy as those 
attending all three years. 


DE usage data and TLC member observations provided indicators of teacher and student use of 
technology for instruction. While the TLC training was not specific to the utilization of DE, DE resources 
were the only digital resources available to all schools early on in the initiative, and DE records were the 
only software data available across the three-year period of TLC. Teachers accessed DE resources more 
often than students. Teacher usage averaged about three days per month in Year 1 and 2 but declined 
to about 1.5 times per month in Year 3. Other resources became available districtwide over time, which 
could explain some of the decline. DE use varied considerably across schools. Due to decreased use over 
time, costs per teacher increased from $39 in Year 1 to $65 per teacher in Year 3. The costs per student 
were about $8 each year. 


Observations of TLC members in a representative sample of schools revealed that teachers utilized 
technology in a variety of ways and at different levels of sophistication to enhance instruction. Teachers 
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most often played a facilitative role (i.e., through coaching and interactive direction of students) rather 
than a lecturing role. Student engagement was also reasonably high. 


Recommendations 


Districtwide professional learning initiatives must be seen as programs designed to impact teachers’ 
practices and ultimately student achievement (Killion, 2008). TLC can inform future district efforts 

based on both strengths and areas for improvement. In terms of strengths, training was of high quality 
with multiple sessions over time, teacher teams were trained together over several years, and data were 
collected on progress made over time. Areas for improvement include: 


e Increasing the chances for strong school implementation by building stronger support from the 
central level (e.g., having an in-house implementation team), reducing the number of 
simultaneous central initiatives, increasing team sizes for larger schools, and including an 
administrator and perhaps an instructional resource teacher on school implementation teams. 


e Monitoring school implementation more closely by providing checklists of “look fors” for 
school observers and considering a teacher-leader role for team members to allocate some of 
their time (e.g., 25%) to coaching and monitoring implementation. Central staff could provide 
guidance on how to manage scheduling so that teachers can visit learning labs or provide 
alternate delivery vehicles for disseminating learning labs (e.g., online video library). 


e Building the likelihood of sustainability with increased coaching as teachers try new practices, 
providing principals with guidance on replacement of team members as needed, and providing 
materials to orient new team members to the effort and its importance. 
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Background 


Evaluation Study 


This is the final report from Data, Research, and Accountability (DRA) on the Teacher Leaders Corps 
initiative (see blue chevron below). 


Status Report Interim Final Report on Years 


on 2013-15 for Documents on 2013-2016 for Staff 
Sielii 2014-2016 for and District Leadership 
Staff 


The study assessed both implementation and outcomes of TLC. Because TLC was rolled out to all schools 
simultaneously, it was not possible to compare technology use and student outcomes over time due to 
the lack of a comparison group. As such, data provided are descriptive in nature (see Table 1). Specific 
conclusions cannot be drawn about TLC’s specific program effectiveness separate from other initiatives 
in the district. Data sources utilized included measures developed by Discovery Education and WCPSS. 
DRA staff utilized training attendance, implementation monitoring surveys, digital integration surveys, 
usage data, interviews, and observations as data sources for the study. Appendix A provides an 
overview of data sources and methods, with additional detail in additional appendices specific to each 
data source. 


Table 1 
Study Design and Supported Conclusions 


Research Design Conclusions that can be Drawn 


OO Experimental We can conclude that the program or policy caused changes in 
outcomes because the research design used random assignment. 
QO Quasi-Experimental We can reasonably conclude that the program or policy caused 
changes in outcomes because an appropriate comparison strategy 
was used. 
VI Descriptive These designs provide outcome data for the program or policy, but 
MI Quantitative differences cannot be attributed directly to it due to lack of a 
M Qualitative comparative control group. 


Sources: List, Sadoff, & Wagner (2011) and What Works Clearinghouse (2014). 


Program Description 


Beginning in 2013, WCPSS Instructional Technology staff contracted with Discovery Education (DE) to 
develop the Teacher Leader Corps (TLC) professional learning initiative. TLC focused on general concepts 
of integration of technology into instruction and was designed to better leverage technology for 
learning. Academics leadership staff chose DE as the training partner because of their expertise and 
capacity to train teams of teachers from all schools simultaneously. Schools were given access to DE’s 
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library of online instructional resources as part of the overall initiative. The training, however, focused 
on general technology integration strategies (i.e., not specifically DE resources). Prior to the start of Year 
1 of TLC (2013-14 school year), principals and other key Central Office staff were introduced to 
Discovery Education, told about the overall goals and expectations of the TLC program, and asked for 
their support. Principals were also required to create a Teacher Leader Corps (TLC) team of four teachers 
per school. Additional information on the history of the initiative is available in a previous report 
(Simmons & Baenen, 2016). 


Figure 1 provides the Pathway of Change for the TLC effort (see Appendix B for a more detailed logic 
model). The DE trainers provided professional learning opportunities through multi-day training in each 
of the three years to the TLC members. TLC members subsequently shared TLC ideas and techniques 
and provided labs to demonstrate lessons in practice with other staff in their schools. Thus, both the 
TLC team members and other teachers in the school were provided with opportunities to learn how to 
incorporate technology more meaningfully into instruction. An increase in appropriate use of 
technology for instruction was then to occur, which would lead to increased student engagement and 
learning. Another goal of the Teacher Leader Corps was to develop leadership skills in members as they 
provided professional learning experiences to their colleagues. 
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Costs 


The TLC initiative was a three-year agreement between WCPSS and DE to provide five full days of 
training per year to four-member teams from each WCPSS school during each academic year. Because 
WCPSS’ schools operate on several different academic calendars, four cohorts were established: 
traditional (elementary), traditional (secondary), modified/single-track year round, and multi-track year- 
round. 


Table 2 provides a yearly breakdown of costs related to the TLC initiative and for Discovery Education 
software licenses. Initially, central school district funds paid for the trainings, substitute teachers, and 
teacher stipends, while schools paid for their individual Discovery Education content licenses. Schools’ 
content licenses were calculated based on the grade span and size of the school. Starting in year 2, 
central school district funds covered all costs related to the TLC initiative including school site licenses. 
Table 3 provides average yearly costs for schools by grade span. As shown in Table 3, prices for licenses 
increased only slightly for elementary and high schools across the years, with an increase at middle 
schools of $577 over time. By year 3 of the initiative, central funds covered the school licenses. 


Table 2 
WCPSS Three-Year Costs for TLC and Discovery Education Initiative 


Year 1 Year 2 Year 3 Overall 
2013-14 2014-15 2015-16 
Discovery Education: = 4549473 $412,020 $420,640 $1,224,033 
School Licenses 
Discovery Education: = 2447 560 $447,500 $447,500 $1,342,500 
Trainings 
Teachers’ Substitutes $202,760 $221,041 $245,323 $669,124 
Teacher Stipends $8,960 $11,694 $1,520 $22,174 
TLC Evaluation by DE $10,000 $10,000 $10,000 $30,000 
Total $1,060,593 $1,102,255 $1,124,983 $3,287,831 


Note: End-of-year figures. 
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Table 3 
Average Costs per School for DE Content Licenses 


Year 1 Year 2 Year 3 
Elementary $2,313 $2,360 $2,391 
Middle $2,313 $2,848 $2,890 
High $2,785 $2,848 $2,890 


Note: Smaller schools paid a reduced rate. 


Research Supporting TLC Initiative 


Technology Adoption 


The adoption of technology and digital media provides many advantages to teachers, students, and 
school districts. Benefits include access to current and timely content, increased interactivity to 
promote student engagement, adaptability to individual student needs, alignment to state and local 
curricular standards, integrated formative assessments, and lower costs over time compared to 
traditional learning modalities (Fletcher, Schaffhauser, & Levin 2012; Johnston 2011; Mardis, Everhart, 
Smith, Newsum, & Baker 2010). 


Instructional technology initiatives, when implemented in a proper and sustainable manner, give 
students access to the tools that they will be expected to utilize effectively after graduating from high 
school. When instructional technology initiatives are properly implemented, research shows significant 
increases in student achievement. A large number of research studies have found student benefits 
within K-12 technology initiatives (Bebell, 2005; Campuzano, Dynarski, Agodini, & Rall, 2009; Goldberg, 
Russell, & Cook, 2003; Hunter & Greever-Rice, 2007; Jeroski, 2003; Lowther, Ross, & Morrison, 2003; 
Shapley, Sheehan, Maloney, & Camikas-Walker, 2010; Suhr, Hernandez, Grimes, & Warschauer, 2010; 
Warschauer, 2006; Weston & Bain, 2010). 


The benefits of instructional technology integration cannot occur if teachers are not comfortable using 
technology or applying it to their lesson planning and instruction. Over time, when teachers begin to 
master available technology, they more consistently implement more media into their lessons. The role 
of the classroom teacher then switches from a lecturer to more of a facilitator role where the teacher 
guides students through learning (Edwards, 2013; Corn, 2009). 


Professional Learning 


Regardless of content focus, effective professional learning that impacts student achievement must 
utilize a research-based model. Joyce and Showers (2002) and Cooper (n.d.) provide excellent 
summaries that point to the following critical components for effective professional development, 
especially for complex efforts: 


1. Developing knowledge through exploring theory and concepts, 
2. Demonstrating or modeling the strategy or skill, 
3. Providing time for initial practice in the workshop to build knowledge and skill, 
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4. Providing coaching, usually by a peer, to build knowledge, skill, and transfer of learning. Peer 
coaching not only contributes to the transfer of training; it involves collaborative planning and 
facilitates the development of new school norms of collegiality and experimentation. 


Of these, the component with the most impact was coaching (Joyce & Showers, 2002). Effective 
professional development is based on a comprehensive overall plan at the district level, occurs in 


multiple sessions over time, and meets the standards for effective professional development (Cooper, 
n.d.). 


District-Level Efforts 


Planned full-day trainings occurred for multiple cohorts. 
Attendance was strong (66% to 71%) in Years 1 and 2, with some 
decline in Year 3 (47%). Participants considered the sessions 
helpful, and 59% of team members participated all three years. 


Did all district-level trainings occur? Was attendance at TLC sessions strong? 


Yes, all district-level trainings occurred with the exception of one cohort of TLC members during Year 2 
who had to complete one of their training days via self-paced virtual modules due to inclement weather 


(i.e., snow). Each of the three years, over 600 members of school-based teams participated in several 
cohort groups. 


Attendance was fairly strong, especially in Years 1 and 2. Appendix C provides more detail. 
e About two-thirds (65.9%) of TLC members attended all trainings in Year 1, and close to three- 


fourths (70.6%) attended all trainings in Year 2. 
e The percentage of TLC attending all five days of training declined in Year 3 (47.2%). 


TLC 2013-14 to 2015-16 DRA Report No. 16.09 


Table 4 
TLC Session Attendance Rates by Year 


Year Attended Attended all # attended 3-4 # attended 1-2 Average Hours of 
at all training days days days PD completed 
# % # % # % 
2013-14 695 459 (66.0%) 189 (27.2%) 44 (6.3%) 29.78 
2014-15 660 466 (70.6%) 139 (21.1%) 55 (8.3%) 30.81 
2015-16 643 303 (47.2%) 294 (45.8%) A5 (6.9%) 29.21 


Source: WCPSS eSchools Databases 
Note: 2014-15 counts for participants attending all days includes participants who did not complete the 5°" day of 


training due to inclement weather. 
What was the attrition rate on TLC teams across years? 


Of the original 695 Year 1 TLC members, 527 continued participation in Year 2, and 407 participated in 
Year 3 to complete their three-year commitment. Thus, over half (58.6%) stayed with TLC all three years, 
but over 40% did not. While program staff felt this was strong, no standard was available for expected 
attrition. These results can serve as a useful benchmark for similar multi-year training models. 


Table 5 
TLC Session Attendance Rates by Year 
Still There — Still There 
2013-14 in 2014-15 in 2015-16 


Started in 2013-14 695 527 407 


% of 2013-14 75.8% 58.6% 


Source: WCPSS eSchools Databases 
Did TLC members fit the guidelines? 


Leadership staff in Academics asked principals to select four classroom teachers to become members of 
the Teacher Leader Corps. Expectations were that teachers would: 


attend five days of training each of the three years, 

integrate technology and develop learning labs within their own classroom, 

work with administrative staff to identify the instructional needs of school staff, 
train other teachers to empower them to use technology within their lessons, and 
coach peers around using this model to improve student learning. 


View hi 


Principals were expected to provide structured time for the TLC participants to deliver effective school 
wide training to teachers, promote the use of digital resources to support common core learning and 
teaching, and provide consistent and on-going feedback to all TLC participants to help improve their 
skills as teacher leaders. 
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As shown below, nearly all participants were teachers (as requested). Other sections of this report 
address training attendance, school training and labs, and administrative support. 


Table 6 
Job Classification of TLC Members (School-Based Only) 
Year 1 Year 2 Year 3 
Job Classification 
N % N % N % 
El 
neal, 248 41.8% 241 44.2% 342 56.6% 
Secondary Teachers 223 37.6% 204 37.4% 175 29.1% 
Teactiers (nen 17.0% 93 17.1% 84 13.9% 
grade level specific) 
Media Coordinators 2 0.3% 2 0.4% 3 0.5% 
School 
saa 1.3% 1 0.2% 0 0.0% 
Administrators 
her Professional 
Ome ms apes 11 1.8% 4 0.7% 0 0.0% 
Total 593 545 604 


Note: School-Based Positions Only 


Lead Trainer Interviews 


At the end of the three-year initiative, DRA staff interviewed five of 
the lead trainers from Discovery Education who had worked with 

different cohorts of schools for at least two of the three years. The 
semi-structured interview gathered descriptive and perceptive data 


“| think Wake has ... [had more] 
central office level presence and 
they’ve had more of the teachers 


about the initiative. Overall, the DE trainers thought that the 

oe. [who] understood the importance of 
training efforts went well. Some of the specific factors that served TLC and that their role was truly 
to facilitate the success of the initiative included the visibility of the important in the district and within 
Central Office staff at trainings, the ability to work consistently with their school.” 


a cohort throughout the initiative, and the opportunity that the TLC 


members had to collaborate with their peers. “| saw in Wake County that my 


schools who had the full support of 
The trainers also discussed some barriers to implementation and their administrators were more 
offered some recommendations for the future. These included a successful than the schools that did 
lack of sufficient administrative support for providing time for TLC not. 
members’ training, labs, and coaching to other school staff. 
Trainers thought the limited coaching they were able to do in 
schools was a valuable aspect of their work. They indicated that 
future DE work with school districts would include more coaching as 
well as talking with principals while in the schools to maintain or 
build support. The Lead Trainer Interview Appendix (Appendix D) provides more detail. 


Lead Trainer 
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School Level Efforts 


During Year 1, approximately 60% of all TLC members had the 
opportunity to share their training modules and demonstrate 
learning labs for their colleagues at their school. These 
percentages dropped sharply in Years 2 and 3 (36% to 43%). 


Did TLC Members Share Training Materials in their Schools? 


During the first year, DE provided TLC members with turnkey training materials following the 
professional learning days, which TLC members could then share with school staff. 
Subsequently, TLC members were given time to develop a presentation during the professional 
learning days. TLC members were sent a survey after each training day to indicate what they 
were able to share at their school and provide feedback; return rates were reasonably strong in 
Years 1 and 3 (56.6% and 50.2%), but lower in Year 2 (36.3%). DRA staff calculated the 
percentage of training or learning lab events (i.e., dissemination events) that TLC members 
reported delivering compared to those they were expected to deliver. See Appendix E for more 
details on methods and results. 


e The percentage of TLC member dissemination events delivered compared to those 
expected was higher in Year 1 (over 60%) than in Years 2 or 3 (36% to 43%). 


e The percentage of TLC member-led trainings delivered to those expected was slightly 
higher in Year 3 compared to Year 2. 
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Figure 2 
Percentage of TLC Expected Dissemination Events that Occurred by School Year 


64.5% 
60.5% 


43.3% 
39.5% 39.2% 


i i : [ 


Learning Lab Training 


@ 2013-14 ™2014-15 m™2015-16 


Source: PD Monitoring Tool data from each school year 
Responses: 2013-14: N=2003 (57.5%, 696 teachers); 2014-15: N=1198 (36.3%, 660 teachers); 
2015-16: N=1613 (50.2%, 643 teachers) (5 data collections after each training day) 


In open-ended responses, the predominant reason TLC members provided for not conducting the 
training was time—not having time on staff meeting agendas because of multiple competing priorities 
and TLC members not having time to prepare a presentation or lab. Table 7 provides additional reasons, 
including “newness” on the team. 


Table 7 
Year 3 TLC Responses to PD Monitoring Question: What prevented you from implementing 
any of the training modules with other teachers at your school? 


Category Frequency Percent 
Time 498 90.9 
New TLC Member 15 2.7 
TLC Member felt unprepared 14 25) 
Lack of support/interest 11 2.0 


Source: PD Monitoring Tool data from Year 3 
Note: Percentage of 548 comments made. Due to rounding percentages may not equal 100%. Comments made in 


less than 2% of the comments are not shown. 
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Did TLCs provide learning labs in their classroom? 


In Year 1 (2013-14), 61% of TLC members were able to conduct learning labs at their school 
sites; however, by Years 2 and 3, that percentage dropped to 39%. Table 8 provides reasons 
given for not conducting learning labs. Within the time category, the following barriers to 
implementing learning labs were identified: 


e administration (9%), 


. : “We are ona strict curriculum 
e testing and/or curriculum mandates (8%), 


which has us teaching lessons as 


e scheduling conflicts (6%), written by CMAPP or Caulkins,” 
e other professional development initiatives (6%), 
° track out (5%), School-wide professional 


. ; development has been focused 
e inclement weather/vacation (3%), an MISS Ter? cling te weaned 


e TLC member absences (2%), and cohort school” 
e too little time between trainings (1%). 
Teacher comments 
___—_—— 

A related reason from another 5% of teachers was a lack of 
interest/participation. This could also relate to time if teachers did not have time or coverage to go to 
the learning labs. Some school administrators provided a substitute to facilitate this. Other common 
reasons from 4% to 5% of respondents related to a lack of technology resources or the TLC member 
feeling unprepared. 


Table 8 
Year 3 TLC Responses to PD Monitoring Question: If no, why have you not 
been able to conduct a learning lab? 


Category Frequency Percent 
Time 430 74.5 
Lack of interest/participation 29 5.0 
Lack of technology resources 27 4.7 
No coverage for teachers 25 4.3 
TLC Member not prepared/lack of training 24 4.1 
New TLC Member 18 3.1 
TLC Member not aware of expectations 10 1.7 
Used other method 7 1.2 


Source: PD Monitoring Tool data from Year 3. 
Note: Percentages are of the 577 comments; Comments made in less than 1% of the comments are not shown. 
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Did teachers increase their knowledge and self-efficacy in terms of integrating technology 
into instruction? 


Teachers’ sense of self-efficacy in utilizing digital resources for instruction 
increased for both the TLC teachers and non-TLC teachers in the school. The 
two groups started out at about the same point, with TLC members 
reaching a higher level after the first year of TLC training and maintaining 
the higher level across the three years. Teachers who were only members of 
TLC for one or two years had similar patterns for self-efficacy as those 
attending all three years. Some slippage in teachers’ sense of self-efficacy 
occurred over the summer months. 


A major short-term goal was for TLC members and later other staff members to increase their sense of 
self-efficacy in appropriately utilizing technology for instruction. Self-efficacy is defined 

as the belief in one’s capabilities to organize and execute the courses of action required to manage 
prospective situations (Bandura, 1986). The Discovery Education Digital Integration Survey (DDIS) 
assesses this in ways consistent with the stages of technology integration identified by Dwyer, Ringstaff, 
and Sandholtz (1990). See Figure 3 and Appendix F for more details. 


Both TLC and non-TLC WCPSS teachers responded to 10 different self-efficacy questions in a DE pre- and 
post-survey. Response rates for TLC members were higher than for non-TLC members given that the 
district has over 10,000 teachers, which may mean that responses were not fully representative of 
teachers districtwide. In fact, the number of responses received from non-TLC members was too low to 
be used in spring of 2014. 
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Figure 3 
Stages of Technology Integration 


Discovery Education Digital Integration Survey (DDIS) 
Stages of Technology & Digital Media Integration 6 <o0i5 orai< 70) 


Transformation 


Adapted from Allsopp, Hohlfeld, and Kemker (2007) and 


Dwyer, Ringstaff, and Sandholtz (1990 (42 < DDIS_Total < 56) 
ie - = Infusion y feacnes 
primarily 
encourage or 
(28 < DDIS_Total < 42) support the 
Adaptation y Students are active 
provided engagement of 
(14 < DDIS_Total < 28) se jeateliles students in 
Adoption Technology : higher-order 
p eaaniniee ait i ixamniog 
(0 < DDIS_Total < 14) integral part of OOls to activities that 
Entr Teachers begin how teachers demonstrate may not have 
y to provide design their their knowledge been possible 
students with lessons to build and make without the use 
Teachers opportunities to students’ connections to of technology 
primarily use use technology understanding of the — 
in conventional content world 


basic skills to 
embed 
technology and 
media into their 
instruction 


ways 


Teacher-Centered Technology Usage Student-Centered Technology Usage 


Note 1: Reprinted from Discovery Education (n.d.). Reprinted with permission. 
Note 2: Discovery Education totals self-rating scores (10 questions with a scale of 1-7) to categorize respondents 
into implementation stages. The ranges above each stage specify the scores assigned to each level. 


Results reflected growth in TLC teachers’ self-efficacy in use of instructional technology over time, with 
some slippage between years (see Figure 4). Figure 4 depicts the percentage of teachers at each stage of 
the Technology and Digital Media continuum based on responses to the DDIS teacher survey. Teachers 
who participated in TLC for one, two, or three years (i.e., varying number of years) are depicted on the 
left; whereas, teachers who experienced the full TLC program (i.e., all three years) are depicted on the 
right. 
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e TLC members showed gains in their movement across the 
Stages of Technology and Digital Media continuum. 

e Some progression was lost in each year’s summer months e.g., 
Spring Year 1 to Fall Year 2) 

e TLC teachers who participated all three years showed gains in 
their movement across the Stages of Technology and Digital 
Media continuum , but those gains did not differ substantially 
from teachers overall, some of whom only participated for one 
or two years. 


Figure 4 


DRA Report No. 16.09 


“| think that a lot of the 
teacher leaders that | 
worked with as time went 
on they were more 
comfortable taking 
instructional risks.” 


Lead trainer 


TLC Teacher Self-Efficacy Over Time in Percentages 


entry ll adoption Mi adaptation Ml infusion ®@ transformation 
TLC Teachers (Varying Number of Years) TLC Teachers (All Three Years) 
a eee - “4 
ee 35.9% 
oe ZA 


Fall 2013 Spring 2014 Fall2014 Spring 2015 Fall2015 Spring 2016 * Fall2013 Spring 2014 Fall 2014 Spring 2015 Fall2015 Spring 2016 


Source: DDIS Survey in fall and spring of each year. 


Note: Only teachers who identified themselves with a valid employee ID on the DDIS, were confirmed at TLC 


teachers by WCPSS, and did not skip items on the survey were included 


Note: 2013-14 Pre n=406, Post n=373; 2014-15 Pre n=506, Post n=206; 2015-16 Pre n=509, 2015-16 Post n=464 


Note: 3 Years n=1,823 
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Note: Percentages at or below 4% are not labeled. 


TLC versus Non-TLC teachers 


For both the TLC and non-TLC groups, self-efficacy averages increased between the pre- and post-survey 
each year for both TLC and non-TLC teachers (see Figure 5). The non-TLC teachers included all other 
teachers in the school district. Both groups started out at the mid-point on the self-efficacy scale, with 
the TLC members reaching higher levels by fall of 2014 and maintaining their advantage past that point. 
However, there was still room for both groups to grow. 


Figure 5 
TLC and Non-TLC Teachers’ Self-Efficacy Over Time 
7 
6 
5 
a 
5 4 
a 
n 
Fg 
a 3 
Les) 
© 
3 2 
. Spri Spri Spri 
pring pring pring 
Fall 2013 2014 Fall 2014 2015 Fall 2015 2016 
—eTLC 3.85 4.94 4.53 5.24 4.57 5.09 
Non-TLC 3.95 4.15 4.51 4.12 4.42 


Source: 2013-14, 2014-15, and 2015-16 DDIS Survey, scale 1-7, does not include N/A responses 

Note: 2013-14 Pre (TLC n=406, Non-TLC n=658); 2013-14 Post (TLC n=373, Non-TLC is missing due to low 
response rate) 
2014-15 Pre (TLC n= 506, Non-TLC n= 1,163); 2014-15 Post (TLC n= 206, Non-TLC n= 2,242) 
2015-16 Pre (TLC n= 509, Non-TLC n= 1,035); 2015-16 Post (TLC n= 464, Non-TLC n= 658) 

Note: Based on a two-sample t-test, differences between TLC and Non-TLC teachers at each time point 
are not statistically different in fall 2013, but are significantly different (p < .001) in fall 2014, spring 
2015, fall 2015, and spring 2016. 
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Was technology incorporated meaningfully in instruction? 


Teachers accessed resources more often per month than students. Teacher usage 
averaged about three days per month in Years 1 and 2 but declined to 1.5 times per 
month in Year 3. Due to decreased use over time, costs per teacher increased from 
$39 in Year 1 to S65 per teacher in Year 3. The costs per student were about $8 
each year. Observations of TLC members in a representative sample of schools 
revealed technology being used in a variety of ways and with varied sophistication. 


DE web usage data and observations provided data relevant to whether or not technology was 
incorporated meaningfully in instruction. 


DE Web Usage Data 


DE was the only instructional technology tool for which usage data was available over the entire three- 
year period of the study, but it should be noted that many other instructional technology resources 
were available to schools, and that the number of these resources increased somewhat across the three 
years. As such, DE usage represents only a sampling of the instructional technology tool usage that 
occurred in schools during the study period. These results can be helpful as the district considers how to 
structure instructional technology contracts and set standards for the level of usage or cost per day or 
per user that justifies purchase of a digital resource. 


Results reveal greater use by teachers than individual students and greater use in Years 1 and 2 than in 
Year 3. Observations supported the greater use by teachers in that teachers would often display video 
or other materials for the full class to view. Whether access rates are appropriate to justify the cost is an 
open question. As shown in Figure 6: 


e Teachers utilized DE an average of about 3 times a month in Years 1 and 2 but about half as 
often (approximately 1.5 times) in Year 3. 
e Students utilized DE about 2 times a month on average in Years 1 and 2 but only 0.3 times per 
month in Year 3. 
Table 9 illustrates that the frequency of use of DE became more consistent (albeit lower) across teachers 
in Year 3 compared to the earlier years. 
e The spread of days per month that teachers accessed DE was much wider in Years 1 and 2 than 
in Year 3 based on standard deviations. 
e Use decreased substantially in Year 3, and this decrease was evident for both TLC and non-TLC 
teachers. 


Additional analyses are shared in Appendix G. Usage varied by month and by school. Many schools 
increased their usage of DE from Year 1 to Year 2. However, no schools increased usage from Year 2 to 
3. No school’s usage rates were higher in Year 3 than in Year 1. 

The fact that usage varied considerably across schools and that use decreased over time could inform 
the structure of district instructional technology contracts in the future. 
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Figure 6 
Average Days per Month Teachers and Students Accessed DE 


m 2013-14 ™ 2014-15 2015-16 


3.39 


1.87 
1:92 


1.50 


0.34 


STUDENT TEACHER 


Source: DE Usage Data (Days per Month) 
Note: Differences over time are statistically significant (p < .001) 
Note: Authentication is defined as a day where the user (student or teacher) accessed DE resources at least once. 


Table 8 
TLC vs Non-TLC DE Days Accessed and Standard Deviations (SD) 


Year 1 Year 2 Year 3 
N Mean SD N Mean SD N Mean SD 
TLC 694 3.07 2.98 610 3.03 2.51 615 2.3 2.19 
Non-TLC 7,340 3.25 2.89 7,346 3.52 2.96 | 6,085 1.31 1.75 
Overall 8,034 3.25 2.89 7,956 3.51 2.96 | 6,700 1.49 1.99 


Source: DE web usage (Authentications) from each year 


Table 10 quantifies the cost of DE resources to WCPSS for students, teachers, and overall. 
Authentication is defined as a day where the user (student or teacher) accessed DE resources at least 
once. Patterns reveal that: 


e The number of student users increased over time, while the number of teacher users decreased. 

e The cost per student user did not vary much over time, but the cost per active teacher 
increased. 

e The cost per day of use increased for both students and teachers and thus overall. The cost per 
day DE was accessed by both groups combined was just over $1. 
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Table 9 
WCPSS Discovery Education Costs and Usage 
Year 1 Year 2 Year 3 
Total Students who Utilized DE 47,961 52,199 52,539 
Total Days Accessed* 221,607 268,722 195,388 
Student Average Days Accessed per Student 4.62 5.15 3.72 
License Costs $391,373 $412,020 $420,640 
Cost Per Active Student $8.16 $7.89 $8.01 
Cost Per Day Accessed (Per Student) $1.77 $1.53 $2.15 
Year 1 Year 2 Year 3 
Total Teachers who Utilized DE 10,001 7,530 6,519 
Total Days Accessed 156,347 142,669 110,163 
Teacher Average Days Accessed per Teacher 15.63 18.94 16.90 
License Costs $391,373 $412,020 $420,640 
Cost Per Active Teacher $39.13 $54.71 $64.52 
Cost Per Day Accessed (Per Teacher) $2.50 $2.88 $3.82 
Year 1 Year 2 Year 3 
Total Users who Utilized DE 57,962 59,729 59,058 
Total Days Accessed 377,954 411,391 305,551 
Overall— Average Days Accessed per User 6.52 6.89 5.20 
Students and License Costs $391,373 $412,020 $420,640 
Teachers Cost Per Active Users $6.75 $6.89 $7.12 
Cost Per Day Accessed (Per User) $1.04 $1.00 $1.37 


Source: Discovery Education Contract (Costs) & DE web usage (* Authentications) from each year 
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Observations 


Observations suggest TLC members utilized technology meaningfully in instruction. 
Information was not available on non-TLC teachers. TLC members played a variety of roles 
across the lessons observed, often two or three roles within a lesson. Roles in which 
teachers facilitated instruction through coaching or interactive direction were more 
common than roles such as lecturing or monitoring. 


Technology was most often considered appropriate, but not necessary, for a lesson. 
Student engagement was also reasonably high. These data suggest that technology was 
used by TLC teachers to enhance instruction. This is an area for further study due to the 
small sample size. 


Methods 


Appendix H provides details about the methodology utilized for teacher observations. With input from 
Instructional Technology and Media Services (ITMS) staff, DRA created an observation tool adapted from 
the International Society for Technology in Education (ISTE) Classroom Observation Tool (ICOT) and the 
Florida Center for Instructional Technology Integration Matrix Scale (Welsh, Harmes, & Winkelman, 
2011). The ICOT is a free online resource for assessing technology integration in classrooms (ISTE, 2009). 
The TIMS Matrix (as reflected in Discovery Education’s DDIS tool) assessed the quality of the interaction 
between the learning environment and the instructional technology integration. DRA added descriptive 
and open-ended questions to gain a more comprehensive understanding of the actual lessons and 
technology’s role in them. 


DRA staff conducted observations of TLC members in the spring of 2016 in a random sample of 10% of 
the district’s schools. The sample was stratified by grade span, and also considered the percentage of 
students who were Educationally Disadvantaged (ED) at each school. Observations of non-TLC teachers 
in the school were not feasible with the resources available. 


Observers first notified principals and subsequently TLC members about the purpose of the observations 
and the week an observer would be at the school. Observers encountered various situations that 
reduced the sample size or influenced scheduling. At some schools, the TLC team did not include four 
members as intended; some teachers were on leave or tracked out, some had student teachers (and 
were therefore not teaching), and some had committee/team obligations that made some days 
impossible for observations. Observers generally observed one lesson, with observations averaging 
about 30 minutes each. Overall, 49 of the 68 observations were completed (72%), as shown in the next 
table. 
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TLC Observations Planned and Completed 


# Pl d # % 

Level/# Schools cine 
Elementary n=11 44 72.7% 
Middle = n=4 16 i - 56.3% 
High n=2 8 100.0% 
Total 68 72.1% 


Note: n specifies number of schools. # specifies number of teachers. 


Teacher Roles 


TLC members utilized instructional technology in a wide variety of ways across classrooms and subjects. 
Examples include showing a video from DE before a discussion of a topic, using an interactive 
spreadsheet to assign students to centers, providing students with several technology resources to 
create group projects, and facilitating student-created web pages or electronic newsletters. 


The Teacher Leader Corps (TLC) observation form asked for the role or roles of the teacher based on five 
categories as shown below. Observers could select more than one role, and often selected two or three 
during the course of one lesson. Teachers most often were observed moderating discussion or providing 
interactive direction (representing over 30% each of the responses). Teachers also facilitated or coached 
in about one fourth of the observations. Observers saw lecturing, modeling, and monitoring students 
least often. Fuller descriptions of each role are included in Appendix H. 


Figure 6 


TLC Teachers’ Role During Lessons Observed 


= Lecturing 

= Facilitate/Coaching 

= Modeling 

= Moderating Discussion 
= Interactive Direction 


= Monitoring students 


Source: TLC Observations. 102 responses were given for 31 of the observations. 
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Observers' notes reflected varied teacher roles—teachers provided individual instruction and 


explanations, served as facilitators of student work, and 
supported students as they worked. More specifically: 


e Individual instruction and explanation of the lessons 
accounted for close to one third of the coded data (10 
responses or 32% of the coded data). For example, in 
one case the teacher discussed the appropriate uses 
of technology and identified good strategies that 
some students were using which could be helpful to 
others. 


e Teachers worked as facilitators in their classrooms 
quite often as well (8 responses or 26% of coded 
data). For example, one observer noted that the 
teacher “facilitates discussion and interaction of 
[students] sharing what they know and adding facts to 
a list." 


e Teachers also functioned in a support role for their 
students fairly often (6 responses or 19% of coded 
data). For example, an observer noted that the 
"teacher and two assistants support[ed] groups of 
students in drafting story problems and coming up 
with a plan. [The] teacher showed examples to the 
students of what they would create (video of her 
students acting out story problem last year). [The] 
teacher support[ed] students in working toward 
creating their own videos." 


Technology Integration Matrix 


The Technology Integration Matrix (TIMS) categorizes lessons 
according to the level of technology integration (i.e., entry, 
adoption, adaptation, infusion, or transformation). Within 
these levels, lessons were further characterized based on the 
learning environment (i.e., active, collaborative, constructive, 
authentic, and goal directed). The full scale is shown in 
Appendix H along with the frequencies with which observers 
saw each standard. Two examples of levels on the technology 
integration scale are shown in sidebars in this section, with 
examples of the other levels shown in Appendix H. 


All levels of technology integration were observed, which is a 


positive finding. Table 12 shows that more lessons were in the 
middle of the matrix than at the extremes. This suggests that 
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TIMS TRANSFORMATIONAL 
SAMPLE 
GRADE 7 


Lesson: The goal of this social studies 
lesson was to develop student 
Wyave(-yacie-Jarellaycae)manelia) o)(em=\ =a] moma) 

Ne alelater-]aM alike) aVAUte mide le (-1almel=\VZ-) (eo) el-1e| 
raaleliaiaatsvelteM elackx-laie-}a le) ace 


Teacher: The teacher briefly lectured 
about several events in American history 
and their impact on the development of 
the United States of today. The teacher 
then assigned each group of four students 
a particular event to research from a pre- 
selected list of resources as well as any 
student-selected credible resources. 


Students: Students worked collaboratively 
within their groups to research their 
assigned event. Utilizing one iPad per 
group, students utilized a nearpod 
software program to create a multimedia 
presentation that explained in detail their 
event, what happened, why it was 

Tan) exelae=laiem-lalemaat=M lanl) (ersjulolaccmn ce) manele 
society. At the end of the project, each 
student group was to present their event 
iol ualomel [-Kcwme(=1iclalomuatsliamanlelialaatexel te] 
presentation, and answer any questions 
from their classmates or the teacher. 


Rationale: This lesson was rated as 
transformational because students were 
actively engaged and had full control of 
their learning once the topic was assigned. 
The teacher served as a facilitator of 
Ret-Taaliayca (alel mma al=mcxo)(=Mcvol0] ger) Mesyab(o(=Tal mom alelo| 
choice of the resources they used and 
collaborated within their team to create 
their presentation. Once completed, 
saUlo(=1avacmvol0] (ol ola=sx-lalar-lalomel-yiclalemual-lia 
research to the class and answer any 
questions. 
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teachers generally used technology in either conventional or advanced ways to provide students 


opportunities to work independently or in collaborative groups. On 
the other hand, TLC members had room to grow in terms of greater 
use of the most advanced levels of technology integration. 


Table 12 
Percentage of Observations in Each Level of Technology Integration 
Entry 10.3% 
Adoption 35.0% 
Adaptation 30.9% 
Infusion 14.4% 
Transformation 9.5% 


As Table 13 shows, average ratings for the level of need for technology 
and the appropriateness of technology were both above the midpoint 
on the five-point scale. The level of engagement was also above the 
midpoint on the scale. Need for digital resources was rated somewhat 
lower than the appropriateness of its use (roughly 3.5 versus 4.3 
across subjects, respectively). Thus, observers often considered 
technology use as appropriate even when it was not necessary for a 
lesson. In that sense, teachers used technology to enhance their 
instruction. 


Results by grade span are shown for descriptive purposes only, 
especially at the middle and high school levels where sample sizes 
were low. This is an area for possible further study. 


Table 13 
Need and Appropriateness of Technology and Student 
Engagement by Grade Span 


Student 
Level (n) Need Appropriateness Engagement 
Elementary (32) 3.4 4.1 4.2 
Middle (9) 3.9 4.8 41 
High (8) 3.4 a 3.1 
Overall (49) 3.5 4.2 4.0 


Source: TLC Observations, Spring 2015. n= number of observations 

Scales: Need for Technology: Not needed (1) to essential (5) 
Appropriateness of Tech: Not appropriate (1) to very appropriate (5) 
Level of Student Engagement: Not engaged (1) to very engaged (5) 


It was not possible to assess the intended student learning or skill 


TIMS INFUSION LEVEL 
GRADE 3 SAMPLE 


Lesson: The goal of this integrated 
language arts and social studies lesson 
was to build student understanding of 
ivalom dal a-tom ole-]aleal-cmelmar-la(e)ar-]| 
government. 


Teacher: The teacher was primarily a 
facilitator and circulated the room 
discussing progress with students. She 
provided students with feedback on 
the content of their graphic organizers. 


Students: Students used web-based 
resources including Discovery 

iso Ulery ola mon (oler-1x-Mlalcelaaar-ialolam-) elel0hs 
the branches of government, which 
they then recorded on a graphic 
organizer. Students selected two 
activities from a tic-tac-toe activity 
sheet, the use of which encouraged 
student-choice. Activities were 
designed to further deepen student 
understanding of the branches of 
government, resulting in the creation 
of two original products using 
technology. As examples, working with 
a partner, students created a 
presentation on Google Slides or Deck, 
created a video clip using Educreation, 
or created diagrams and graphics. 
Students worked on laptops, desktop 
computers, iPads, and tablets. Students 
collaboratively used technology, 
sometimes employing multiple devices 
to create original products. 


Rationale: This lesson was rated as 
infusion because (a) the tic-tac-toe 
activity sheet encouraged flexible, self- 
directed student choice and (b) group 
work fostered collaboration in 
meaningful activities. 


outcomes for TLC due to the lack of a comparison group and the possible impact of multiple initiatives 


simultaneously. 
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Effective Professional Learning for Implementation and Sustainability 


The impact of staff development efforts is of key interest nationwide. Killion (2008) points out that 
isolated staff development sessions have little chance for impact. On the other hand, professional 
learning programs are ongoing, coherent, and linked to student achievement. Regardless of content 
focus, effective complex professional learning that improves student learning must provide not only 
content but modeling, initial practice, and coaching (Joyce & Showers,2002; Cooper, n.d.). Cooper (n.d.) 
also points to research that indicates effective professional development is based on a comprehensive 
overall plan at the district level, occurs in multiple sessions over time, and meets the standards for 
effective professional development. WCPSS utilized Learning Forward (n.d.) standards as they planned 
this professional learning initiative. 


TLC was a strong professional learning initiative, incorporating many research-based practices. These 
included having one team per school, meeting many times over three years, setting expectations for 
team members and principals that were aligned with educational standards, providing quality resources 
and learning experiences with opportunities for practice, and gathering data in a variety of ways all 
represent best practices in professional learning (Cooper, n.d., Learning Forward, n.d). TLC was strong in 
terms of setting expectations and providing some materials for school implementation, but sharing of 
training at the school level by TLC members was variable (in large part due to time constraints). Delivery 
of school-wide training and use of key resources dropped off in Year 3. Perhaps the weakest aspect of 
the TLC initiative was in the coaching, at both the central and school level. This component is considered 
critical for the transfer of learning to teachers’ every-day practice (Joyce & Showers, 2002). Some TLC 
members received coaching from DE trainers if their school purchased an additional science component 
of the software, but most did not. In turn, TLC members’ coaching of other teachers in their school was 
generally limited because of regular classroom duties. 


The implementation science frameworks developed by NIRN (n.d.) are research-based and focus closely 
on whether efforts are rolled out in a way that promotes scaling up a large initiative across sites. While 
there is some overlap with the professional learning standards and other research discussed above, a 
general assessment of features of TLC that promoted scale-up in schools and those which could be 
strengthened can inform future professional learning efforts in WCPSS. The three types of 
implementation drivers identified by NIRN as beneficial are listed below, with a general assessment of 
WCPSS’ status on each. 


Table 14 
Implementation Drivers / Ratings 
Implementation Drivers Rating 
Competency—Training and Coaching of TLC Members Strong 
Organization—Central and school support structures Fair 
Leadership—Support for technical and adaptive issues Fair 


Possible Ratings: Very Strong, Strong, Fair, Weak 


Competency Drivers: 
1. Were the right staff utilized as trainers and trainees? 
2. Was training of high quality to build competency and enable implementation? 
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3. Did staff have a chance to practice the new skills with others? 
4. Was coaching available from the district level? 


TLC rated high for the first three questions as described above. The amount of coaching available, 
however, was fairly limited. 


Organization Drivers (central and school support structures): 
1. Did WCPSS support the intervention with adequate resources, time, etc.? 
2. Wasa district implementation team available to discuss how to handle issues that arose? 
3. Was the administration in the school supportive? Did schools have an implementation team to 
guide and monitor implementation? 
4. Did staff collect data on implementation? Did staff review data promptly to influence 
implementation? 


Organizational drivers were not as strong as the Competency area. 


At the district level, WCPSS supported the intervention with adequate professional learning (13 full days 
over three years), but a district implementation team was not in place for ongoing support beyond these 
days to address questions or challenges that arose. In addition, all TLC members did not receive the 
resource of 10 iPads as originally planned. A variety of data were collected (e.g., training attendance and 
satisfaction, pre- and post-surveys on teacher efficacy in using digital resources, school implementation 
surveys, usage of DE) but analysis and utilization were slower than would be desirable. 


At the school level, most school administrators were supportive, but the level of support varied. 
Generally, the less supportive the principal, the less likely time was provided for presentations to staff or 
learning labs scheduled in a way that non-TLC teachers could observe. The school-based TLC team 
supported implementation as best they could, but these teams fell short of the full facilitative role 
described in implementation science. Monitoring TLC implementation, coaching, and trouble-shooting 
technical issues were not part of the expectations for the team and were only generally mentioned for 
administrator roles. Survey data was collected about staff perceptions of the training and presentations 
to staff, but use of these data, along with direct observation of technology use, was not emphasized in 
the initiative. 


Leadership Drivers: Was support available to address both technical issues (supplies, resources, 
schedules, etc.) and adaptive issues (making adjustments as issues arise)? 


Limited district support was available for technical and adaptive issues, largely due to funding 
limitations. 


Some schools reported having an insufficient number of devices or a lack of a sufficient internet 
connection to properly take advantage of digital resources. WCPSS needed to hire DE to deliver the 
training because of the limited capacity of the small district instructional technology team. As such, the 
district’s internal capacity to support the ongoing work was stretched too thin. Whether DE staff 
provided support on the content of the training between sessions is unknown. School-based TLC 
members provided content support on a limited basis as time allowed. Some schools did not replace TLC 
members after they left, which also reduced the likelihood the effort would take hold long-term. 
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TLC’s strengths and areas for improvement should both be considered for future digital resource 
acquisitions as well as future professional learning initiatives in WCPSS as described below. 


Consider Potential Use Before Purchasing Digital Resources 


While some digital resources are free, others are a major expense. Results of the analyses on Discovery 
Education resources have implications for future resource procurement. Before purchases are made, 
district and school staff should ask questions such as: 


e What need will this resource help us to meet? Do we need this resource in addition to what we 
already have? Should we discontinue use of another product because this one meets our needs 
better? 

e What do we expect to accomplish as a result of the use of this resource? How can we check to 
see if this actually occurs? What will we do when we get the expected results (or not)? 

e Will teachers be the primary users or will students need frequent direct access? 

e How often do we expect staff or students will utilize this resource? How much are we willing to 
pay for that level of use? How can we check utilization levels and how often? 


Answers to these questions can help us select the best resources to meet our needs and to negotiate 
the best possible contracts for their use. School and district staff may want to work through these 
questions together to create some guidelines for all to consider in making such purchases. Using the 
district’s purchasing power to select resources that can be efficiently supported from the district level 
will also help. 


Consider Goals and Buy In Before Launching District Professional Learning 


Launching too many “good” initiatives at once with school staff as the primary audience can result is 
inconsistent buy-in and implementation by schools. TLC was not the only significant districtwide 
professional development effort that occurred during this time span, which could have hampered full 
implementation. Reducing the number of simultaneous initiatives, making the relationships clear 
between initiatives when more than one is happening, and coordinating scheduling can all build buy-in 
from school staff. A district implementation team could serve this role (which could be an existing team 
or a separate one). As an example, this year’s professional learning focused on The Vision 2020 Strategic 
Plan integrates initiatives more fully and delivers training on existing meeting days; these are steps in 
the right direction. 


More specific suggestions for ways to build buy-in with school leadership could include alerting them to 
the plans for the coming year, soliciting their input on the initiatives more fully, including an 
administrative member on school implementation teams, and communicating frequently regarding the 
status of the effort and expectations. (NIRN, n.d.) 
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Monitor school implementation more closely 


NIRN (n.d.) found scale-up of interventions was much more likely with implementation teams in place at 
each level (e.g., schools and district) to facilitate rolling out any initiative, monitoring its success, and 
addressing challenges that arise. Smaller district implementation teams (e.g., with project staff) can 
provide ongoing support for initiatives, with periodic updates to the larger district team. One support 
that could be helpful is the development and sharing of videos of exemplary lessons in a virtual library at 
the district level. Area superintendents and their staff can also monitor implementation through their 
contacts with schools. 


At the school level, TLC teams fulfilled part of this implementation role, mostly through presentations 
and learning labs. School implementation teams could be strengthened by including an administrator. 
This member could facilitate scheduling of sharing opportunities and monitoring of implementation. In 
addition, this person could help develop and then convey “look-fors” to others on the administrative 
team. For example, with TLC, the learning labs provided a useful way for teachers to see demonstrations 
of instructional technology in action. However, some schools did better than others in setting schedules 
that allowed teachers to visit TLC members’ classrooms. One successful strategy was to have a 
substitute teacher who moved from classroom to classroom so that teachers could observe the lab 
lesson. 


Monitoring implementation at regular intervals and promptly following up with low implementing 
schools or teachers could also help determine possible barriers and supports to address. 
Implementation team meetings should be maintained long after the training ends to monitor status and 
ensure practices become ingrained into daily instruction. New practices take time to become part of 
normal practice. 


The drop in the frequency of TLC members sharing practices learned at the school level suggests a 
possible drop in interest and/or implementation for the initiative. This is common as the “newness” of 
an initiative wears thin and new initiatives begin. Ways to maintain or rejuvenate enthusiasm for 
initiatives should be planned up front, under the assumption that enthusiasm may wane over time. Staff 
might consider this as a topic for discussion in annual meetings, periodic web announcements, or posts 
on social media about school or teacher successes. 


Build the Likelihood of Sustainability 


The work of Joyce and Showers (2002) points to the critical role of coaching in securing fidelity of 
implementation for new initiatives. Peer coaching is encouraged and more coaching for TLC members as 
well as for other teachers in the school setting as they practiced new strategies would have 
strengthened the TLC training model. Classroom teachers, which nearly all TLC members were, had 
limited time to coach others in their school. If funds allow, creating a teacher-leader role to free up 
some of their time (perhaps 25%) to coach and monitor implementation would be extremely helpful. If 
this is not feasible, an instructional coach might play this role. 


TLC was designed to be a three-year commitment for team members. While 60% of TLC team members 
participated all three years, 40% did not. All initiatives need to have plans to address the inevitable 
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replacement of team members and school administrators. TLC data suggest members’ skills and sense of 
efficacy with technology improved whether participation was for three years or less, so it may be one- 
or two-year efforts can often be adequate in length. 


Overall, TLC was a strong professional learning initiative which incorporated many research-based 
practices. WCPSS tends to be strong in selecting staff to involve in initiatives and providing high quality 
training—TLC was no exception. TLC was stronger than some other training models utilized in WCPSS in 
utilizing a teacher team per school that was trained simultaneously; in providing clear expectations for 
follow-up, and in supplying some of the resources to TLC members to share with school staff. 
Establishing district and school support structures to facilitate implementation could increase the 
chance that practices learned will become part of daily instruction. 
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Appendix A 


Evaluation Methods and Sources 


Year 1 = 2013-14 


Year 2 = 2014-15 Year 3 = 2015-16 


Data Source/ When 
é Notes 
Appendix Collected 
Attendance at Years 1,2,3 Session Attendance: Electronic files were provided by 
Trainings June each year Professional Learning which reflected attendance for each 
(App. C) team member for each of the five days of training each 


year. This was cross-referenced with DE files for usage 
analyses. 


Training Topics: Available from eSchools and project 
records. 


Year 1 Completed 
after each day; 
gathered in June 


Training Evaluations: Training attendees were required to 
complete a short satisfaction survey based on the trainings 
they received in Year 1. This requirement was removed in 


each year Year 2, and return rates were very low. We requested that 
staff change this back in Year 3, but it did not occur, and 
return rates were again too low to use. Therefore, this data 
is available only for Year 1 of the effort. 
Lead Trainer Year 3 DRA staff conducted 6 interviews with the district 
Interviews Summer coordinator of the TLC initiative (in person) and a sample of 
(App. D) DE trainers (by phone). Interviews were semi-structured 
and focused on their reflections of the initiative over the 
three years. 
DE PD Years 1,2,3 This DE survey addressed whether or not TLC members 
Monitoring Tool Deployed after provided trainings to staff members at their schools or 
(App. E) each day of TLC conducted learning labs at their school since the last day of 
training TLC training. 
DE Digital Years 1,2,3 DE deployed the DDIS survey at the beginning and ending 
Integration Beginning of year of each school year to both TLC and non-TLC members. The 
Survey (DDIS) (pre) and end of DDIS survey measured teacher’s self-efficacy with 
(App. F) year (post) integrating technology and digital resources into their 


classrooms. DRA included the questions in the district 
survey for Non-TLC members to increase return rates and 
representativeness of the sample. 


DE Usage Data Year 1,2,3 Data on individual usage of DE were collected via DE’s 
(App. G) Continuous online portal for every WCPSS teacher and student (log- 
ons, types of digital resources accessed (e.g., videos, lesson 
plans), etc. 
Observations Year 2,3 In Year 2, observations focused on best practices for 


(App. H) 


nominated teachers. In Year 3, a random sample of TLC 
teachers were observed to see level of utilization of 
technology and student engagement. 
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Appendix B 


Logic Model for TLC Initiative 


Need: Wake County had insufficient availability and use of digital resources to support instruction. Technology was used to support 
instruction inconsistently. Thus, one need was to broaden teachers’ toolkits. In addition, WCPSS was spending too much money on 
separate school-based licenses for Discovery Education (DE). The contract was re-worked at the district level over three-years 
beginning in 2013-14. DE provided access to DE resources for all schools and is providing PD in how to utilize technology resources. 


OUTCOMES — IMPACT 
Short-Term Intermediate 
2013-14 2014-15 2015-16 


increased teacher w 
- full training program on DE usage rates over 2014- 
resources will be available (3:1). Digital-Age leaming by ani i 15 
resources 


iscovery Education Resources and 2.Train four-teacher teams from every Higher student 
others. school were recruited in 2013-14 to Teacher Leader Corps (TLC) engagement in 
receive 5 full days of staggered (6-8 prov i classrooms in TLC vs. 

Central staff time to coordinate weeks apart) professional ini i ; i i non-TLC classrooms. 

trainings development on the use of DE ogy y Higher engagement in 
resources i 7 high volume vs. low 

Discovery Education Trainers i volume users (based 
3. Provide a district license for all i on usage and teacher 

Teacher Substitutes students and teachers for discovery icipati survey results) 
education and other resources. 


While not measurable 


in an increase in use of DE 
Digital Age resources by 
¢ TLC teachers 
¢ Non-TLC 
teachers 
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Appendix C 


Attendance at Trainings 


Methods 


Instrument and Data Collection 

WCPSS tracks teacher participation in professional development using the eSchools professional 
learning management system. In order to attend trainings, teachers were enrolled in DE professional 
development and their attendance was recorded using eSchools. As such, data regarding participation in 
DE professional development was downloaded from the eSchools interface. 


Results 


As a participating member of TLC, teachers were required to engage in five professional development 
sessions during each school year, resulting in fifteen total sessions per TLC member. Due to the large 
number of teachers participating in TLC, teachers attended trainings in groups of approximately 30 to 40 
teachers. During 2013-14 (Year 1), DE trainers delivered 166 sessions, which included mostly in-person 
group professional development, as well as some individual in-classroom coaching to TLC participants. 
During 2014-15 (Year 2), DE trainers delivered 105 sessions to TLC members. During 2015-16 (Year 3), DE 
trainers delivered 105 sessions to TLC members. One group of teachers missed the fifth session in Year 2 
(2014-15) due to inclement weather. These teachers were asked to complete the session on their own 
with provided materials and support from DE trainers, but their completion of these materials was not 
tracked. 


Table C.1 depicts the participation rates and the percent of TLC teachers who attended all or some of 
the professional development sessions. Participation in professional development was relatively strong, 
with 66% and 71% of the TLC members attending all days of the training in 2013-14 and 2014-15, 
respectively. On average, TLC members received about 30 hours of training per year. 
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Table C.1 
TLC Session Attendance Rates by Year 
Year Attended Attended all # attended 3-4 #attended1-2 Average Hours 


at all trainings days days of PD 

# % # % # % completed 
2013-14 695 459 (66.0%) 189 (27.2%) 44 (6.3%) 29.78 
2014-15 660 A66 (70.6%) 139 (21.1%) 55 (8.33%) 30.81 
2015-16 643 303 (47.2%) 294 (45.8%) AS (6.9%) 29.21 


Source: WCPSS eSchools Databases 
Note: 2014-15 counts for participants attending all days includes participants who did not complete the fifth day of 
training due to inclement weather. 


What was the attrition rate on TLC teams across years? 


Table C.2 shows the attrition rate for TLC teachers from years one to two and years one to three. There 
were 695 TLC teachers in 2013-14, 646 in 2014-15, and 626 in 2015-16 (as of October 2015). Out of the 
695 TLC teachers in 2013-14, 527 were still TLC participants in 2014-15 and 407 in 2015-16. 


Table C.2 
TLC Session Continuous Enrollment Rates across Years 
Still There — Still There 
2013-14 in2014-15 in 2015-16 
started in 2013-14 695 527 407 
% of 2013-14 75.8% 58.6% 


Did TLC members fit the guidelines? 


Principals were asked to select four teachers from their building who showed potential to become 
teacher leaders. Principals were encouraged to select classroom teachers to engage in TLC, and they 
were asked to not select technology contacts, media specialists, IRTs, or other with formal coaching or 
professional development responsibilities. Table C.3 depicts the job classifications for TLC teachers 
during Years 1, 2, and 3. Nearly all TLC teachers fit the description of elementary, secondary, or non- 
grade level specific teachers. During Year 1, 21 participating teachers (3%) did not fit the specified 
criteria for teachers; in subsequent years, less than one percent of participating teachers did not meet 
the classroom teacher criterion. Additional detail regarding TLC teachers’ job classifications is listed in 
Tables C.5 through C.7. 
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Table C.3 
Job Classification of TLC Members (School-Based Only) 
Job Year 1 Year 2 Year 3 
Classification N % N % N % 
Elementary 
Teach 248 41.8% 241 44.2% 342 56.6% 
eachers 
Secondary 
Teach 223 37.6% 204 37.4% 175 12.4% 
eachers 
Teachers (non- 
grade level 101 17.0% 93 17.1% 84 13.9% 
specific) 
Media 
CaeCinnot 2 0.3% 2 0.4% 3 0.5% 
School 
ai 8 1.3% 1 0.2% 0 0.0% 
Administrators 
Other 
Professional 11 1.8% 4 0.7% 0 0.0% 
Staff 
Total (School- 
Based Positions 593 545 604 
Only) 


Source: eSchools Database and WCPSS Employee Database. 
Did TLC members believe the training was of high quality and helpful? 


Following the professional development sessions, teachers were asked to complete a survey in eSchools 
regarding the quality of the DE professional development. Teacher survey data was collected during 
Year 1 (2013-14), but due to a technical issue, the professional development satisfaction survey was not 
required in Years 2 and 3. Therefore, data on teacher satisfaction with the training itself was only 
collected for Year 1. Table C.4 depicts teacher satisfaction with the TLC trainings following the 2013-14 
trainings. During the first year, 86% or more of teachers agreed with the statements supporting the 
helpfulness of the DE professional development sessions. 
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Table C.4 
TLC Member eSchool Ratings of 2013-14 Training 
% 
Item 

Agree 
The facilitator(s) created a respectful and inclusive environment for my learning. 94.9% 
The training/learning objectives were clearly identified in a structured agenda. 92.6% 
The training/learning objectives clearly matched the course description in eSchools. 91.5% 
The facilitator(s) clearly connected the course content to current research/relevant data 90.0% 
sources. — 
The training content clearly built on my prior level of knowledge/skills. 88.5% 
The facilitator(s) gave me adequate time to collaborate with others. 88.4% 
The facilitator(s) gave me adequate time to reflect on how | will use this learning in my 87 A% 
classroom/at my job. ie 
If | implement what | learned from this training it will significantly enhance the effectiveness 36.9% 
of my work in my classroom/on my job. ve 
This training helped me develop strategies to make instruction more relevant for diverse 36.3% 
. 0 


learners. 


Note: N=611 Sorted by % Agree (Descending) 
Source: 2013-14 eSchools session ratings. 
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Table C.5 


2013-14 TLC Member Characteristics 


Job Classification 2013-14 


Frequency Percent 


Assistant Principals (Non-Teaching) 
Elementary Teachers 

Guidance 

Media- Audiovisual Staff 

Other 

Other Professional Staff 

Other Teachers 

Principals 


Secondary Teachers 


Teachers (Federally Funded) 
Elementary 


Teachers (Federally Funded) Secondary 
Teachers (Locally Funded) Elementary 
Teachers (Locally Funded) Secondary 
Teachers (State Funded) Elementary 
Teachers (State Funded) Secondary 


Teachers (Vocational Funded) 


Source: eSchools Database and WCPSS Employee Database. 


Al 


221 


0.86 


31.80 
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Table C.6 
2014-15 TLC Member Characteristics 

Job Classification 2014-15 Frequency Percent 
Assistant Principals (Non-Teaching) 1 0.15 
Elementary Teachers 211 32.66 
Media- Audiovisual Staff 2 0.31 
Other 1 0.15 
Other Classroom Teachers-State 1 0.15 
Funds 
Other Professional Staff 3 0.46 
Other Teachers 92 14.24 
Secondary Teachers 121 18.73 
Teachers (Federally Funded) 2 0.31 
Elementary 
Teachers (Federally Funded) 1 0.15 
Secondary 
Teachers (Locally Funded) Elementary 9 1.39 
Teachers (Locally Funded) Secondary 4 0.62 
Teachers (State Funded) Elementary 119 18.42 
Teachers (State Funded) Secondary 78 12.07 
Teachers (Vocational Funded) 1 0.15 


Source: eSchools Database and WCPSS Employee Database. 


Table C.7 
2015-16 TLC Member Characteristics 

Job Classification 2015-16 Frequency Percent 
Elementary Teachers 229 35.67% 
Media- Audiovisual Staff 3 0.47% 
Other 8 1.25% 
Other Professional Staff 8 1.25% 
Other Teachers 98 15.26% 
Secondary Teachers 105 16.36% 
Teachers (Federally Funded) 3 0.47% 
Elementary 
Teachers (Locally Funded) Elementary 9 1.40% 
Teachers (Locally Funded) Secondary 4 0.62% 
Teachers (State Funded) Elementary 107 16.67% 
Teachers (State Funded) Secondary 66 10.28% 
Teachers (Vocational Funded) 2 0.31% 


Source: eSchools Database and WCPSS Employee Database. 
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Appendix D 
Lead Trainer Interviews 


Methods 


Data Collection 

At the end of the three-year TLC initiative, evaluation staff interviewed five of the lead trainers from 
Discovery Education who had worked with different cohorts of schools over the past three years. The 
interview protocol consisted of a semi-structured interview to gather descriptive data about the DE 
trainers’ involvement with the initiative, as well as questions to elicit their perceptions of their overall 
analysis of the process. Each individual interview lasted approximately 10 to 20 minutes. 


Data Analysis 

District staff members transcribed and then read through all responses to develop initial codes from the 
data. In all, there were 87 coded comments. The initial codes were then reorganized and integrated 
into larger themes that emanated from the interviews. This second coding cycle resulted in five themes 
and nineteen subthemes. 


Results 


Majority of the DE trainers had worked with the initiative for all of the three years. Overall, they 
described the feedback from the TLC members throughout that time period as positive. During the 
interviews, DE trainers referenced specific factors that served to facilitate the success of the initiative 
including the visibility of the Central Office staff at trainings, being able to consistently work with a 
cohort throughout the initiative, and the opportunity that the TLC members had to collaborate with 
their peers. 


They also discussed some barriers to implementation and offered some recommendations for going 
forward. These included trying to better secure administrator buy-in for the project, emphasizing the 
coaching piece where the DE trainers would have been able to work with individual TLC members in 
their cohort at their school sites, and making the expectations more clear in the beginning of the 
initiative so as to aid recruitment of TLC members. 


The trainers also discussed their perceptions of professional outcomes demonstrated by the TLC 
members that they worked with throughout the initiative. Overall, the DE trainers noted that they felt 
the TLC members grew as professional leaders in their schools. In particular, they gained the ability to 
lead collaboration efforts at their schools. In addition, the DE trainers expressed that they felt the TLC 
members developed the potential to lead future instructional initiatives at the school level. 


43 


TLC 2013-14 to 2015-16 DRA Report No. 16.09 


Table D.1 
Frequency of TLC DE Trainer Interview Responses. 


Theme Code/Comments Frequency (%) 


Positive Feedback about the initiative a) 


10) 
Barriers/Challenges to Implementation aS ta ty) 


0 
Recommendations a) 


0 
TLC Member Outcomes 25 (29.1%) 


Feedback from TLC members was 
positive. 
Note: All five DE trainers mentioned these themes. 


5 (5.8%) 


Table D.2 
DE Trainer Interview Responses 


Examples/Quotes 


Positive Feedback —e The visibility of the Central Office staff at e The presence of Central Office staff at 

about the Initiative trainings helped clarify the goals and reinforce district trainings helped make the teacher 
the importance of the initiative to TLC understand “the importance of TLC and that 
members. their role was truly important in the district 

e Consistency of DE trainer working with the and within their school.” 
same cohort all three years helped build e §=“I think that having the consistency of one 
rapport with TLC members. presenter with a specific group the whole 

e Teachers liked the opportunity to collaborate three years really helped to build a nice 
with colleagues. relationship where the teachers were able 

e Schools with principal support experienced to trust me. They believed the things | was 
more successful implementation. presenting to them.” 

e Focusing on pedagogical strategies before the © “..once we gota feel of how much material 
technology led to more success and buy-in should be presented giving them time to 
from the teachers. have that hands on time to plan and work 

e Coaching provided important supports to together and collaborate, they truly loved 
members. Thea 

e The design of the training balanced inquiry e “Wake County did an awesome job putting 
and modeling and kept teachers engaged. the training in front of the devices.” 


e “Sol met with them either ten to twelve 
times a year and worked with their four 
teachers they had selected for TLC and we 
did just a process of just meeting, observing, 
and coming up with individual goals for each 
teacher, and then we did a cycle of co- 
teaching and modeling and then also 
inviting other teachers from the school into 
their classroom to observe and that went 
very well.” 
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Examples/Quotes 


Barriers/Challenges 
to Implementation 


Recommendations 


TLC Member 
Outcomes 


e TLC member turnover rates made consistency 
difficult among cohorts. 

e Some of the expectations and end goals of the 
initiative were unclear to TLC members at the 
onset. 

e Lack of administrator support at some schools 
adversely affected the implementation. 


e DE Trainers felt more coaching opportunities 
were needed. 

e Increased administrator buy-in is critical. 

e Revise the method of TLC member selection. 


e TLC members became more comfortable 
taking and helping others take instructional 
risks. 

e TLC members learned how to lead 
collaboration efforts at their schools. 

e TLC members developed the skills to become 
instructional leaders. 

e TLC members gained the ability to 
differentiate to meet the learning needs of the 
teachers at their schools. 

e TLC members grew professionally. 

e TLC members developed the potential to lead 
other instructional initiatives at their schools. 
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e “A couple of the concerns that the teachers 
had was the turnover in their group.” 


e “It would be helpful to clarify more to the 


teachers when they would be receiving their 
devices and what the timeline was for that.” 


e = One trainer commented that in particular, 


re-delivery of the skills learned were not 
possible considering “their administrator 
didn’t provide the allotted time to go back 
and teach their peers.” 


e Two of the interviewees raised the point that 
more coaching opportunities would be 
beneficial to the initiative. 

e A frequent comment from the trainers 

reflected that getting more administrator buy- 

in is critical to future success. “So if there was 
that administrator piece where the 
administrators really bought into the TLC and 
really understood what was happening on the 
teacher level and the growth that could 
happen then | think that could really enhance 
it and the teachers would find it more valuable 
as well.” 

Three of the DE trainers felt that if the County 

had devised a better method to choose its TLC 

members, participants’ investment in the 
program could be enhanced. “But if there was 

a way that they could apply so that they were 

a little bit more invested so the presenter 

didn’t have to create that level of investment 

and begin the initial piece with that, it would 
have been great.” 


Two of the trainers indicated they saw their 
teachers become “more comfortable taking 
instructional risks” over time. 

Three of the trainers saw their cohort 
members leading collaboration efforts at their 
own schools. “Sharing, collaborating, so they 
started leading other teachers and sharing and 
collaborating.” 

e Some of the teachers were seen as gaining the 
skills to differentiate and meet the individual 
staff needs of their colleagues at their own 
schools. “...so they just became more 
comfortable just leading their staff as a whole 
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Topics Findings Examples/Quotes 
and recognizing what each individual teacher 
might need...” 

e Teacher professional growth was cited as an 
outcome four times during the five interviews. 
“.,. they had grown as a teacher, as an 
individual, throughout the training process.” 

e There were six comments throughout the five 
interviews where the trainers referenced the 
potential that the TLC members had to 
continue to lead instructional initiatives at 
their schools. “I see them being an excellent 
resource for when the teachers, for when the 
schools, are doing professional development.” 
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Appendix E 
DE PD Monitoring Tool 


Background 


TLC members were expected--as part of their initial commitment to the program--to re-deliver 
important aspects of each day’s training at their schools. TLC members were also expected to conduct 
learning labs in which they would open their classrooms for other teachers at the school to give them an 
opportunity to observe some of what they learned through the TLC trainings. School leaders were asked 
to provide support (e.g., planning time, classroom coverage, presentation opportunities) for TLC 
members to accomplish these tasks after each completed day of TLC training (approximately five times 
per year). 


Methods 


Data Collection 

Discovery Education developed an informal survey instrument--the PD Monitoring Tool (see link at 
bottom of this page), to measure the frequency with which teachers completed the expected 
dissemination as well as their perceptions of what went well or did not go well. This survey consisted of 
a combination of multiple choice, Likert rating scales, and open-ended items. The PD Monitoring Tool 
was launched by DE after each of the five training days each year of the initiative. Each TLC participant 
was sent an email with a link to the online survey. In Year 1, 56.6% of the participants who attended the 
trainings completed the survey. In Year 2, the response rate fell to 36.3% of the TLC attendees and in 
Year 3 the response rate increased to 50.2%. 


Table E.1 
Survey Return Rates Per Year 


Year Number TLC Number of Number of Actual | Response Rate 
Members Potential Submissions 
Submissions 
2013-2014 695 3480 2003 56.6% 
2014-2015 660 3300 1198 36.3% 
2015-2016 643 3215 1613 50.2% 
Analyses 


Frequency counts from the PD Monitoring Tool were tallied by district staff across all of the days. 
Percentages of dissemination events completed were also calculated. If teachers had indicated on the 
survey that they did not conduct a learning lab or training module, the survey then asked for the TLC 
members to elaborate on why they were unable to conduct these events. This response was collected 
as an open-ended item. 


https: 
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For those TLC members who had been prompted to report why they had not conducted a training or 
hosted a classroom learning lab since the last day of the TLC training, district staff analyzed and coded 
their open-ended responses for overall themes (see later under Results, Training Modules) to provide a 
clearer picture of what obstacles had prevented them from disseminating these events at their schools. 
The number and percentage of comments related to each theme was then calculated. For the most 
common responses (e.g., time), subcodes were devised and analyzed as well to better characterize the 
nature of the time constraints. 


Results 


Did TLCs share training with school staff? 

Results indicated that 65% of the TLC members who completed the PD Monitoring Tool were able to 
provide training modules for the teachers at their school during Year 1 (2013-14). By Year 2 (2014-15), 
this number dropped by roughly one third to 36%. In Year 3 (2015-16), the percentage of TLC members 
who were able to deliver the training modules increased slightly to 43%, see Figure E.1. 


Did TLCs set up a learning lab in their classroom? 

In Year 1 (2013-14), 61% of TLC members conducted learning labs at their school sites; however, by Year 
2 (2014-15), that percentage dropped to approximately 40%. In Year 3 (2015-16), the percentage 
remained relatively consistent with Year 2 with 39% of the TLC members conducting learning labs. 
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Figure E.1 
TLC Dissemination Events by School Year 


64.5% 


60.5% 


43.3% 


39.5% 39.2% 


Learning Lab Training 


35.5% 


™ 2013-14 ™2014-15 m™2015-16 


Source: PD Monitoring Tool data from each school year 
Responses: 2013-14: N=2,003; 2014-15: N=1,198; 2015-16: N=1,613 
Note: Data were collected at five different time points after each training. 


Table E.2 
Year 3 TLC Responses to PD Monitoring Question: What prevented you from implementing any of the 
training modules with other teachers at your school? 


Category Frequency Percent 

Time 498 90.9 

New TLC Member 15 2.7 

TLC Member not prepared 14 2.5 
Lack of support/interest 11 2.0 

Not enough information to present 5 0.9 
Alternative plan/method 4 0.7 
Attrition 1 0.1 


Source: PD Monitoring Tool data from Year 3 
Note: Due to rounding percentages may not equal 100. 
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Training Modules 

Time was the most cited challenge that prevented teachers from implementing a training module at 
their school, comprising 90% of the 548 responses (see Table E.2). A subcode analysis of the 498 “time” 
responses resulted in ten subcodes: 


e general reference to time (32% of 498), 

e administration did not allot time (20% of 498), 

e other professional development initiatives (16% of 498), 

e inclement weather/vacation*, 

e track out*, 

e schedule conflicts*, 

e testing and/or curriculum mandates*, 

e no professional development or early release days on the schedule*, 
e TLC member absences*, and 

e not enough planning time or class coverage*. 


Note: * indicates frequency was less than 10%. 


As shown in Table E.2, The TLC members cited other factors that did not relate to time; however, they 
occurred at a much lower frequency from 0.1% to 2.7% of the total responses. The next most frequent 
response (2.7%) was made by new members of the TLC program with comments such as, “This is my 
first session.” Approximately 2.5% of the TLC members responded with quotes such as “What’s a 
training module?” or “First year on the team” which seemed to indicate the TLC member was not 
prepared to redeliver training. Two percent of the responses conveyed that the TLC members failed to 
hold redelivered training at their school due to a lack of interest or support among staff with quotes 
such as “Lack of teacher motivation,” and “We attempted to do an online training but received no 
response from our staff.” 
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Why have you not been able to conduct a learning lab? 
For those who provided a response for why they had not conducted a learning lab at their school since 
the last TLC district training, 577 responses were analyzed and coded into 10 categories, see Table E.3. 


Similar to the TLC members’ responses as to why they were unable to redeliver training to their schools, 
time was also cited as the largest factor impeding their ability to conduct a learning lab at their school. 


Table E.3 
Year 3 TLC Responses to PD Monitoring Question: If no, why have you not been able to conduct a 
learning lab? 


Category Frequency Percent 
Time 430 74.5 
Lack of interest/participation 29 5.0 
Lack of Technology Resources 27 4.7 
No coverage for teachers 25 4.3 
TLC ae) a las es 54 Hel 
New TLC Member 18 3.1 


TLC Member not aware of 


expectations Ag tel 
Used other method 7 1.2 

Lack of support 5 8 
Previous unsuccessful attempts 2 3 


Source: PD Monitoring Tool data from Year 3 


Of the total 74.5% of responses that related to time, 35% of those responses made only a general 
reference to “time” with statements such as “We have not had time.” To better understand the 
remaining 4% of time related responses, district staff conducted a subcode analysis of these responses 
that resulted in 9 subcodes for the Time category: 
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e general reference to time (35%), 

e administration (9%), 

e testing and/or curriculum mandates (8%), 

e schedule conflicts (6%), 

e other professional development initiatives (6%), 
e track out (5%), 

e inclement weather/vacation (3%), 

e TLC member absences (2%), and 

e too little time between trainings (1%). 


As can be seen above, TLC members felt that testing and curriculum mandates, other professional 
development initiatives, and schedule conflicts at their schools adversely affected the time they had 
available to implement a learning lab at their school similar to why they were unable to redeliver the 
training. For example: 


“We are ona strict curriculum which has us teaching lessons as written in CMAPP or Caulkins.” 
“School-wide professional development has been focused on MTSS Tier 2 plans, as we are a 
cohort school.” 


In addition to time prohibiting TLC members from conducting learning labs, 24.5 % of responses made 
up the remaining 9 codes. As depicted in Table E.3, some of the more frequently cited challenges were a 
lack of interest by staff members (5%), a lack of appropriate technology resources for students and/or 
teachers to allow them to successfully conduct a learning lab (5%), the inability to obtain coverage for 
teachers’ classrooms to enable them to observe in the TLC members’ classrooms (4%), and feelings of 
unpreparedness on the part of the TLC members (4%). 


Four percent of the responses comprised the remaining four codes. See Table E.3 for a complete listing 
of the codes and their respective occurrences. 


Summary 


As can be seen from the results, many of the reasons given by TLC members for being unable to 
disseminate trainings at their schools were similar to the reasons stated for being unable to conduct 
learning labs. For both events, responses related to time constraints were the most frequent comprising 
91% of the total responses for the inability to redeliver trainings and almost 75% for not conducting 
learning labs. 
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Appendix F 


DE Digital Integration Survey (DDIS) 
Methods 


One short-term goal of the TLC initiative was to increase TLC teachers’ self-efficacy in utilizing 
technology appropriately for instruction. Another goal was for TLC teachers to cultivate other teachers’ 
self-efficacy in utilizing instructional technology, thereby increasing all teachers’ self-efficacy. Self- 
efficacy is defined as the belief in one’s capabilities to organize and execute the steps necessary to 
successfully accomplish tasks in a particular domain (Bandura, 1986). 


Data Collection 

Discovery Education (DE) deployed the DDIS to all teachers in the district (TLC and non-TLC members) at 
the beginning and at the end of each school year throughout the three-year project (2013-14, 2014-15, 
and 2015-16). The method of dissemination varied by group (i.e., TLC vs. non-TLC) with TLC members 
completing the survey on the first and last day of training each year. Non-TLC members, all WCPSS 
teachers excluding TLC members, were invited to complete the DDIS survey via email. The 2013-14 Post 
test for non-TLC members was deployed during the last week of the 2013-14 school year with a three- 
day survey window and produced a very low response rate (12 participants). Due to this low response 
rate, DRA managed and deployed the 2014-15 and 2015-16 non-TLC DDIS Post survey as part of the 
WCPSS annual teacher survey. This resulted in much higher response rates. Table F.1 provides response 
rates for both TLC and non-TLC teachers for each survey deployment. 


Response rates were higher for TLC teachers. 

e Response rates for TLC teachers ranged from 54% to 78% in all but one case. The exception was the 
2014-15 post-survey, which had a lower return rate (32%). Late dissemination and a short window 
for completion contributed to the low return rates in this case. Thus, results for spring 2015 may be 
less representative of the overall group of TLC members. 

e Non-TLC teacher response rates were low overall, which limits the generalizability of the findings for 
non-TLC teachers and comparisons to TLC members over time. 
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Table F.1 
DDIS Teacher Response Rates 
TLC Non-TLC 
Time point eee Population i Horch Population zs 
responses : response | responses : response 

size (N) size (N) 
(n) rate (n) rate 
2013-14 Pre 406 695 58.4 658 9,177 7.2 
Post 373 695 53.7 12 9,177 0.1 
2014-15 Pre 506 647 78.2 1,163 9,329 12.5 
Post 206 647 31.8 2,242 9,329 24.0 
2015-16 Pre 509 662 76.9 1,035 8,392 12.3 
Post 464 662 70.1 658 8,392 7.5 


Source: DDIS survey data, human resources data, and TLC participation for 2013-14, 2014-15, and 2015-16. 
Note: DDIS survey data were matched to human resource data and TLC participation information using employee 
identification numbers. Only teachers who completed all 10 DDIS are included as respondents. 


Matching Employee Identification Numbers to DDIS Data 

Prior to analysis, individual teacher responses on the pre- and post-surveys were matched based on 
teacher’s employee identification number to teacher characteristics, which were made available by 
WCPSS human resources department. In some cases, teachers provided incorrect employee 
identification numbers. Teachers’ responses that were not able to be hand-matched to information 
from the human resources department using email address or other identifying information were 
excluded from analysis. This excluded less than 15% of responses at each survey time point. Based on 
district records, the number of years that a teacher participated in the TLC initiative was included. 
Teachers were only included as a TLC teacher in the analysis if the system had a record of them 
participating in TLC. 


Instrument 

The Discovery Education Digital Integration Survey (DDIS) assesses teachers’ self-efficacy toward 
instructional technology in ways consistent with the stages of technology integration identified by 
Dwyer, Ringstaff, and Sandholtz (1990). The DDIS is based on the University of Florida’s Technology 
Integration Matrix (Allsopp, Hohlfeld, & Kemker 2007) and references the five levels of technology and 
digital media integration cited by Dwyer et al (1990) as shown in Figure F.1. These stages represent a 
range in the ways teachers integrate instructional technology for learning, spanning from basic use (i.e., 
entry) to high-level engagement in learning that would not be possible without technology (i.e., 
transformation). To illustrate, a teacher at the entry stage of self-efficacy toward technology may reach 
his or her limit of comfort in using technology at basic use, which might include having students practice 
multiplication facts using a website. On the other end of the spectrum, a teacher at the transformation 
stage of self-efficacy toward technology is comfortable having students share their learning with others 
in a virtual space via student-created websites or podcasts. In this way, students have autonomy in their 
technology use and the use of technology supports their learning in ways that would not be possible 
without it. 


The DDIS includes 10 items that specifically target teachers’ self-efficacy with technology integration. 
The items on the survey are written such that higher scores reflect stages of self-efficacy that are more 
aligned with the transformational technology integration. The 10 items were scored on a scale 
specifying 1=Not True of Me Now and 7=Very True of Me Now. Total scores were generated for each 
teacher across the 10 items; teachers were then assigned to a stage of technology integration based on 
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cut scores established by Discovery Education. The 10 items are included in Table F.2 and descriptive 
statistics are reported in Table F.3. Means for each item are depicted in Table F.3. Overall, these means 
indicate that both TLC and non-TLC teachers increased on each item within each year as measured by 
the pre- and post-survey. 


Table F.2 
DDIS Survey Items 


Question 


lam skilled at working on my own to identify digital resources and media (such as 
videos, interactive reading passages, and simulations) that are aligned with my 


uestion 1 : : 
“ state standards and to integrate them into my lesson plans and student 
assignments. 
lam skilled at using digital resources and media to assign classroom activities and 
Question 2 homework that are tailored to students’ unique learning styles, working strategies 


and abilities. 


lam skilled at working with students as they use the classroom technology and 
Question 3 digital resources to locate, analyze, evaluate and use information to support their 
research and learning. 


lam skilled at engaging my students to participate in online collaborative projects 
Question 4 (not including email exchanges) with other students or professionals to work on 
their self-selected projects. 


lam skilled at locating and implementing instructional units that emphasize 
Question 5 students using the classroom technology and digital resources to solve “real- 
world” problems or issues. 


| am skilled at developing instructional activities that allow students to create 
Question 6 their own web pages or multimedia presentations to showcase what they have 
learned in class. 


| am skilled at using digital resources to develop formative and summative 


uestion 7 : . 
S assessments aligned with the content standards. 


| am skilled at engaging my students in collaborative learning activities that 
Question 8 allow them to work with other students in the classroom to clarify their 
conceptual understanding and to engage in creative thinking and planning. 


lam skilled at presenting information to students using multimedia 
Question 9 presentations or electronic “slideshows” to reinforce the content standards 
that | am teaching. 


| am skilled at developing instructional activities that allow students to use 
Question 10 digital models and simulations to explore complex systems and issues, identify 
trends and forecast possibilities, process data, and report results. 
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Figure F.1 
Discovery Education Digital Integration Survey (DDIS) Stages of Technology and Digital Media 
Integration 


Discovery Education Digital Integration Survey (DDIS) 
Stages of Technology & Digital Media Integration 66-015 soraie 70) 


Transformation 
Adapted from Allsopp, Hohlfeld, and Kemker (2007) and 
Dwyer, Ringstaff, and Sandholtz (1990) (42 < DDIS_Total < 56) Seaches 
Infusion ‘4 primarily 
encourage or 
(28 < DDIS_Total < 42) support the 
Ada ptation Students are ects 
provided engagement of 
(14 < DDIS_Total < 28) sc uated a aisles in 
: 4 Technolo selecting an tha 
Adoption heiebebete A using technology —— 
(0 < DDIS_Total < 14) integral part of tools to activities that 
Ent ‘4 Teachers begin how teachers demonstrate may not have 
sia to provide design their their knowledge been possible 
students with lessons to build and see ; without the use 
Teachers opportunities to students’ sera ‘ rey ° of technology 
primarily use use technology understanding of e id =Ice 
basi sldlle tis in conventional content wor 
embed wae 


technology and 
media into their 
instruction 


Teacher-Centered Technology Usage Student-Centered Technology Usage 


Note 1: Reprinted from Discovery Education (n.d.). Reprinted with permission. 

Note 2: Discovery Education totals self-rating scores (10 questions with a scale of 1-7) to categorize 
respondents into implementation stages. The ranges above each stage specify the scores assigned to each 
level. 
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Results 
Sense of Self-Efficacy with Technology 


TLC Member Responses 


The goal was for TLC members to increase their self-efficacy over time in using technology meaningfully 
for instruction. Therefore, the percentages of staff who considered themselves to be at the entry or 
adoption level over time decreased and those who considered themselves to be at the infusion or 
transformation level increased over time. Levels of technology integration were assigned to teachers 
based on their sum score on the 10 items on the Discovery Education pre- and post-survey. The cut 
scores for each stage are specified in red in Figure F.1. As Figure F.2 illustrates, the percentage of TLC 
teachers who reported self-efficacy at the entry or adoption stages with technology decreased each 
spring from 3% in 2014 to 2% in 2015 and 1% in 2016 (i.e., the percentage of teachers who scored the 
lowest levels each spring). Within year, the percentage of teachers who viewed themselves at the 
transformation level increased each spring, going from 6% to 30% in 2013-14, 20% to 41% in 2014-15, 
and 22% to 36% in 2015-16. The increases in fall 2014 and 2015 of teachers who rated themselves as 
entry or adoption is likely attributed to staff turnover across years (23% between 2013-14 to 2014-15 
and 40% between 2013-14 to 2015-16). Principals were asked to select new TLC teachers when teachers 
withdrew from the initiative or moved to different schools. Given this influx of new teachers each year, 
comparing results from fall to spring within each year is best practice. 


Based on all respondents to the pre- and post-surveys, trends in results support the attainment of the 
goal of increased self-efficacy during each year of program implementation. As shown in Figure F.2: 


e From fall 2013 to spring 2014, the percentage of TLC teachers rating themselves at the entry or 
adoption level decreased from 20% to 3%, and the percentage rating themselves at the infusion 
or transformation level increased substantially from 42% to 73%. 

e From fall 2014 to spring 2015, the percentage of TLC teachers rating themselves at the entry or 
adoption level decreased from 8% to 2%. The percentage rating themselves at the infusion or 
transformation stage increased from 61% to 77%. 

e From fall 2015 to spring 2016, the percentage of TLC teachers who rated themselves at the 
infusion or transformation stage increased from 62% to 78%. 
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Next results were disaggregated by those teachers who were part of the initiative for all three years 
compared to teachers who were part of the initiative for one or two years (Figure F.4). Here we focus on 
the results for teachers who were part of the TLC initiative for all three years and therefore received the 
full TLC treatment. Figure F.2 depicts levels of self-efficacy for teachers who were part of TLC for all 
three years. For teachers who were in the program for all three years, the percent of teachers reporting 
their self-efficacy toward technology integration at the highest level (i.e., transformation) increased 
from 5% in fall 2013 to 40% in spring 2016. Trends in levels of self-efficacy for teachers who were 
engaged in TLC for one or two years are illustrated in Figure F.4. 


Figure F.2 
TLC Teacher Self-Efficacy Over Time in Percentages 


Bi entry Bl adoption i adaptation BB infusion ) transformation 
TLC Teachers (Varying Number of Years) TLC Teachers (All Three Years) 
6.1% | | | f 5.1% 


19.9% 21.6% 19.2% 


23.9% 


29.9% 


Fall 2013 Spring 2014 Fall2014 Spring 2015 Fall 2015 Spring 2016 ~— Fall 2013 Spring 2014 Fall 2014 Spring 2015 Fall2015 Spring 2016 


Source: DDIS Survey in fall and spring of each year. 

Note: Only teachers who identified themselves with a valid employee id on the DDIS, were confirmed at TLC 
teachers by WCPSS, and completed the 10 DDIS items are included 

Note: 2013-14 Pre n=406, Post n=373; 2014-15 Pre n=506, Post n=206; 2015-16 Pre n=509, 2015-16 Post n=464 

Note: 3 Years n=1,823 

Note: Percentages at or below 4% are not labeled. 
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TLC Versus Non-TLC Members 


Figure F.3 shows average self-efficacy scores for both TLC and non-TLC teachers for the years when data 
were available. Response rates were too low (0.1%) for the 2013-14 post-test to be able to provide a 
trend comparison for non-TLC teachers. For both the TLC and non-TLC groups, self-efficacy averages 
increased between the pre- and post-survey each year for both TLC and non-TLC teachers by about the 
same amount. TLC teachers as a group started out higher, but the expectation that their increase would 
be more rapid was not met. Teachers’ level of self-efficacy was based on their response to the Discovery 
Education pre- and post-survey. Table F.3 lists the 10 self-efficacy questions from the Discovery 
Education pre- and post-survey, as well as means for each question across years. 


Figure F.3 depicts that overall TLC and non-TLC teachers followed similar trends in their self-efficacy over 
time. As shown in Figure F.3, eachers felt most efficacious at the end of year compared to the beginning 
of year. Only 12 non-TLC teacher responses were recorded in spring 2015; therefore, an examination of 
self-efficacy for non-TLC teachers from fall 2013 to fall 2014 is not appropriate. The means in fall 2013 
indicate that TLC and non-TLC teachers reported similar self-efficacy toward technology integration. The 
response rates for the non-TLC teacher survey were low; therefore, the findings are not generalizable to 
all non-TLC teachers. 


As shown in Figure F.3, both TLC and non-TLC teachers’ average self-efficacy scores increased between 
the pre- and post-test for Years 2 and 3 of the initiative. It is interesting to note that although it was 
expected that the self-efficacy of the TLC group would increase at a more rapid rate than the non-TLC 
group, both groups in Years 2 and 3 experienced relatively the same amount of growth between the pre- 
and post-tests. Based on mean scores, TLC and non-TLC teachers reported the highest mean self-efficacy 
in spring 2015. Average self-efficacy decreased slightly in spring 2016 compared to spring 2015, whereas 
we would have hoped that the score would have increased consistently across time. 
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Figure F.3 
TLC and Non-TLC Teachers’ Self-Efficacy Over Time 
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—eTLC 3.85 4.94 4.53 5.24 4.57 5.09 
—— Non-TLC 3.95 4.15 4.51 4.12 4.42 


Source: 2013-14, 2014-15, and 2015-2016 DDIS Survey, scale 1-7, does not include N/A responses 

Note: 2013-14 Pre (TLC n=406, Non-TLC n=658); 2013-14 Post (TLC n=373) 
2014-15 Pre (TLC n= 506, Non-TLC n= 1,163); 2014-15 Post (TLC n= 206, Non-TLC n= 2,242) 
2015-16 Pre (TLC n= 509, Non-TLC n= 1,035); 2015-16 Post (TLC n= 464, Non-TLC n= 658) 

Note: Based on a two-sample t-test, differences between TLC and non-TLC teachers at each time point are not 
statistically different in fall 2013, but are significantly different (jp < .001) in fall 2014, spring 2015, fall 2015, 
and spring 2016. 


Teacher self-efficacy across time points for teachers who were part of the initiative for one, two, or 
three years are depicted in Figure F.4. While the percentage of third year TLC teachers (i.e., 3 Years) who 
reported that their level of self-efficacy was in the infusion or transformation stage was 80%; the 
percentage of teachers who were only involved in TLC for one year (i.e., 1 Year) was 72%, which does 
not provide strong evidence that participating in TLC across all three years was associated with 
substantially higher self-efficacy. It is possible that self-efficacy increased for all teachers because the 
TLC teachers shared the instructional technology resources with other teachers at the school, increasing 
the distributed expertise for all teachers in the district. 
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Figure F.4 
Teacher Self-Efficacy Over Time By Number of Years Participating 
Bi entry I adoption BB adaptation BB infusion © transformation 
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Source: DDIS Survey in fall and spring of each year. 

Note: Only teachers who were matched to human resource data are included. 
Note: 1 Year n=545; 2 Years n=559; 3 Years n=1,823 

Note: Percentages at or below 3% are not labeled. 


Table F.3 depicts the means for each item on the Discovery Education pre- and post-survey at each 
timepoint. 
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TLC 


DRA Report No. 16.09 


Self-Efficacy Questions from Discovery Education Pre and Post-Survey 


Item 


Non-TLC 


1.1 am skilled at 


Pre 


2013-14 


2014-15 


Post Pre 


Post 


2015-16 


Pre 


Post 


2013-14 


Pre 


2014-15 


Pre 


2015-16 


Pre 


Post Post Post 


identify digital 
(such as videos, 


passages, and 


standards and to 


lesson plans and 
student assignments. 


2. 1am skilled at using 


working on my own to 
resources and media 
interactive reading 


simulations) that are 
aligned with my state 


integrate them into my 


4.66 


5.69 | 5.30 | 5.80 


5. 


29 


5.62 


4.67 | 5.55 | 4.72 | 5.03 | 4.80 | 5.04 


digital resources and 
media to assign 
classroom activities 
and homework that 
are tailored to 
students’ unique 
learning styles, working 
strategies and abilities. 


3. lam skilled at 


3.67 


4.91 


4.45 | 5.32 | 4.63 


5.11 


3.87 | 5.18 | 4.19 | 4.59 | 4.19 | 4.59 


working with students 
as they use the 
classroom technology 
and digital resources to 
locate, analyze, 
evaluate and use 
information to support 
their research and 
learning. 


4.1am skilled at 


4.24 | 5.22 


4.80 | 5.54 | 4.82 


5.29 


4.37 


5.27 | 4.49 | 4.79 | 4.46 | 4.76 


engaging my students 
to participate in online 
collaborative projects 
(not including email 
exchanges) with other 
students or 


3.12 


4.23 


3.88 


4.76 | 4.02 


4.59 


3.29 


4.64 | 3.57 | 4.13 | 3.51 | 3.84 
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Group TLC Non-TLC 
Item 2013-14 2014-15 2015-16 2013-14 2014-15 2015-16 
Pre Post Pre Post Pre Post Pre Post Pre Post Pre Post 


professionals to work 
on their self-selected 
projects. 


5. lam skilled at 
locating and 
implementing 
instructional units 
that emphasize 
students using the 
classroom technology 
and digital resources 
to solve “real-world” 
problems or issues. 


3.54 | 4.85 | 4.31 | 5.12 | 4.38 | 5.00 | 3.71 | 5.18 | 4.02 | 4.44 | 3.96 | 4.33 


6.1 am skilled at 
developing 
instructional activities 
that allow students to 
create their own web | 3.27 | 4.51 | 3.98 | 4.84 | 4.03 | 4.68 | 3.34 | 5.00 | 3.53 | 3.96 | 3.43 | 3.77 
pages or multimedia 
presentations to 
showcase what they 
have learned in class. 


7. 1am skilled at using 
digital resources to 
develop formative 

and summative 3.69 | 4.72 | 4.29 | 5.09 | 4.43 | 4.95 | 3.79 | 5.00 | 4.00 | 4.37 | 4.08 | 4.33 
assessments aligned 
with the content 
standards. 


8.1 am skilled at 
engaging my students 
in collaborative 
learning activities that 

allow them to work | 4.18 | 5.24 | 4.83 | 5.34 | 4.73 | 5.20 | 4.19 | 5.45 | 4.50 | 4.67 | 4.32 | 4.56 
with other students in 
the classroom to 
clarify their 
conceptual 
understanding and to 
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Group 


TLC 


Non-TLC 


Item 


2013-14 
Pre Post 


2014-15 
Pre Post 


2015-16 
Pre Post 


2013-14 2014-15 2015-16 
Pre Post Pre Post Pre Post 


engage in creative 
thinking and planning. 


9.1 am skilled at 
presenting 
information to 
students using 
multimedia 
presentations or 
electronic 
“slideshows” to 
reinforce the content 
standards that | am 
teaching. 


10. 1am skilled at 
developing 
instructional activities 
that allow students to 
use digital models and 
simulations to explore 
complex systems and 
issues, identify trends 
and forecast 
possibilities, process 
data, and report 
results. 


4.98 | 5.63 


3.21 | 4.42 


5.45 | 5.86 


3.95 | 4.80 


5.39 | 5.75 


4.02 | 4.70 


4.96 | 5.82 | 4.98 | 5.10 | 4.88 | 5.12 


3.23 | 5.09 | 3.53 | 4.00 | 3.44 | 3.87 


Note: Blue Shade/font Indicates post-survey values, which all showed an increase over the pre-test. 
Note: Scale is 1=Not true of me now and 7=Very true of me now (O=N/A not included in descriptive statistics). 
Note: Pre was given in the fall before the first training and post in the spring after the last training. 
Note: Only teachers who identified themselves with a valid employee id on the DDIS, were confirmed as TLC 
teachers by WCPSS, and completed the 10 DDIS items are included. 
Note: 2013-14 Pre (TLC n=406, Non-TLC n=658); 2013-14 Post (TLC n=373) 
2014-15 Pre (TLC n= 506, Non-TLC n= 1,163); 2014-2015 Post (TLC n= 206, Non-TLC n= 2,242) 
2015-16 Pre (TLC n= 509, Non-TLC n= 1,035); 2015-16 Post (TLC n= 464, Non-TLC n= 658) 
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Appendix G 


DE Usage Data 


Methods 


One short-term goal was for TLC members (all WCPSS teachers long-term) to use digital resources more 
frequently during instruction. Discovery Education resources are one type of digital resource that 
teachers and students were encouraged to use in their classrooms. For the purpose of this evaluation, 
DE is the only resource that DRA staff had the ability to track frequency of use for students and teachers. 
Therefore use of other technology resources is unknown. Instructional Technology staff did indicate that 
some other technology resources were introduced to staff over time, which may help explain some of 
the trends found at least partially. 


The Discovery Education digital resource platform tracks the frequency with which teachers and 
students log in to the resources based on teacher or student ID number. DE refers to each day a user 
logs in to the DE resources as an authentication. The duration of the authentication was not recorded. 
The data reported in this appendix were collected throughout each school year for the TLC initiative. 


Results 


How often did TLC and non-TLC teachers access DE resources? Did usage increase over time? 

Figure G.1 depicts the average frequency of TLC and non-TLC teacher authentications per month. In the 
first and second years of the initiative, TLC and non-TLC teachers accessed the DE digital resources about 
three times per month. In the third year of the initiative, TLC and non-TLC teachers used the DE 
resources approximately once or twice per month on average, indicating a decrease in usage in the last 
year. Non-TLC members actually accessed DE slightly more in Years 1 and 2 but less in Year 3. No specific 
goal was set for the frequency of usage desired, so it is difficult to say whether this level of use is 
sufficient or cost effective. 
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Figure G.1 
TLC vs Non-TLC Teachers Average Monthly Authentications 
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Source: Discovery Education Usage Data (Days per Month) 
Note: 2013-14 (TLC n=694, Non-TLC n=7,340) 

2014-15 (TLC n= 610, Non-TLC n= 7,346) 

2015-16 (TLC n= 615, Non-TLC n= 6,085) 


Table G.1 shows the average monthly authentications and standard deviation for TLC and non-TLC 
teachers. The standard deviations indicate more variation in teacher use in the first and second years of 
the initiative than in the third year. Thus, the standardized variance around the mean was close to three 
days in Years 1 and 2 versus less than two and % days in Year 3 (ranging from 1.31 for non-TLC teachers 
to 2.3 for TLC teachers). This suggests that teachers were much more consistent in using DE at a lower 
level in Year 3. 


Table G.1 
TLC vs Non-TLC Teacher Average Monthly Authentications with Standard Deviations 
Year 1 Year 2 Year 3 
N Mean SD N Mean SD N Mean SD 
TLC 694 3.07 2.98 610 3.03 2.51 615 2.3 2.19 
Non-TLC 7,340 3.25 2.89 | 7,346 3.52 2.96 6,085 1.31 1.75 
Overall 8,034 3.25 2.89] 7,956 3.51 2.96 6,700 1.49 1.99 


Source: Discovery Education Usage Data (Days per Month) 
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Goal: Use of technology increases over time 


Another goal of the TLC initiative specified that teacher and student appropriate usage of technology 
resources should increase across time. This includes the usage for all students and teachers in WCPSS. 
Again, data was only available for DE to measure this goal. Figure G.2 illustrates student and teacher 
average monthly authentications. The figure shows that teacher and student usage was consistent from 
the first to the second year of the initiative. In the third year of the initiative, usage decreased. The 
average monthly authentications for students and teachers ranged from about twice per month for 
students to about three times per month for teachers during the first two years (with decreased usage 
in the last year). This evidence suggests that teachers and students were not using DE resources more 
frequently across time as desired. However, it is possible that teachers were utilizing other digital 
resources and therefore not using DE as frequently. 


Figure G.2 
Student and TLC and Non-TLC Teacher Average Monthly Authentications 


m2013-14 2014-15 2015-16 


Average Montly Authentications 


Student Teacher 


Source: DE Usage Data (Days per Month) 


Figure G.3 illustrates student and teacher average authentications by month for each year of the 
initiative. The figure shows that students consistently used DE resources less often than teachers. 
Average usage was lower in the third year of the initiative relative to the first and second years. Use 
tended to be highest in October for teachers. 


67 


TLC 2013-14 to 2015-16 


Figure G.3 
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Student and Teacher Average Authentications by Month 


School Year Month 
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Note: Insufficient data was available in June to generate an average. 


Table G.2 depicts the overall monthly average authentications for teachers and students by grade span 
including elementary, middle, and high schools. 

e Elementary school teachers and students had the highest overall average monthly usage during 
each year of the initiative. Middle and high school teachers and students were almost similar in 
their average monthly usage. 

e Average monthly usage for all grade spans did not exceed three authentications per month, 
suggesting infrequent usage. Additionally, all grade levels decreased in their usage from the first 
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and second year to the last year. In the first year, average monthly usage ranged from 1.8 for 
high schools to 2.6 for elementary schools; in the third year, average usage was 0.3 for middle 
and high schools to 0.7 for elementary schools. 


Table G.2 
Average Authentications for Teachers and Students (combined) by Grade Span 
School Year 
Grade Span 2013-14 2014-15 2015-16 
Elementary 2.58 2.53 0.66 
Middle 1.94 1.96 0.29 
High 1.79 1.94 0.29 


Source: DE Usage Data 


Table G.3 depicts teacher and student average monthly authentications and the percent difference in 
average authentications from the previous year by grade span including elementary, middle, and high 
schools. Again, the table shows that in the first year, elementary students and teachers used the DE 
resources most often. For elementary students, average monthly usage of the DE resources decreased 
from 2.1 in the first year to 0.5 in the third year. For elementary teachers, average monthly usage of the 
DE resources decreased from 3.6 to 1.9 in the third year. 
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Table G.3 
Student and Teacher Average Authentications and Change from Previous Year by Grade Span 
Role Grade Span 2013-14 2014-15 2015-16 
Elementary ; 
Avg. Authentications 2.08 2.08 0.45 
Students Middle 
Avg. Authentications 1.76 1.82 0.25 
High School 
Avg. Authentications 1.41 1.53 0.19 
Elementary 
Avg. Authentications 3.62 3.78 1.87 
Teachers Middle 
Avg. Authentications 2.62 2.88 0.94 
High School 
2.06 2.62 0.59 


Source: DE Usage Data 


Note: E indicates elementary schools; M indicates middle schools; H indicates high schools. 


Table G.4 depicts the average student and teacher monthly authentications by area or region of WCPSS. 


Teacher and student monthly usage is disaggregated by elementary, middle, and high schools. Overall, 


this table shows that student and teacher usage was similar in all areas of the district, including 


elementary support schools. Patterns across years mirrored those found in other analyses. Students 
used DE resources about twice per month and this average declined in the third year to less than one 
authentication per month. Teachers used DE resources between two to three times per month during 
the first year, with a decline in the third year of less than twice per month in most regions of WCPSS. 
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Table G.4 
Average Student and Teacher Monthly Authentications by Area 


2013-14 2014-15 2015-16 
E M H E M H E M H 
Central Student 2.28 tet ucieg| Resiesilene as ees peas 
Teacher 3.90 2.32 2.04 404 2.82 2.23 1.86 0.89 0.55 
Student 1.98 1.74 1.74 1.99 1.80 1.69 0.44 0.50 0.18 
Eastern | t 
Teacher 3.68 2.87 1.88 3.67 2.65 2.29 1.81 0.77 0.66 
Student 2.08 2.03 0.40 
Elem. Support 
Teacher 3.53 3.33 1.23 


Student 1.83 2.56 148 1.96 2.14 1.39 041 0.27 0.19 


Northeastern 
Teacher 3.72 2.49 2.60 3.89 2.33 3.32 1.83 0.87 0.53 
Student AAS) as} PD || al AL XO) 1.49 | 0.44 0.29 0.19 
Northern | — } 
Teacher 3.32 2.55 2.04 3.69 2.62 2.29 1.85 0.87 0.58 
Student Ais} sy) asi) || afore} aL 7/5} 1.57 | 0.45 0.18 0.22 
Northwestern - : ; rf 
Teacher 3.94 2.35 2.14 4.00 2.61 2.76 2.15 0.58 0.45 
Student Papp iNhes——AU-EXO). |] alist? aL 7/5} 1.30 040 0.14 0.17 
Southern | f 
Teacher 3.56 2.61 2.04 3.69 3.06 3.36 1.86 1.14 0.90 
Student Poi ANey7/ ae! || Wale) aL 7/7/ 1.74 |} 041 0.18 0.18 
Southwestern t t ~ } 
Teacher 3.47 2.59 1.86 3.77 2.98 2.31 2.13 0.88 0.45 | 
Student TL aLSs7h algal || ail Aas} dP4A0R OS 20229) 0.14 
Western | 


Teacher 3.47 2.85 2.08 3.84 3.23 2.81 2.04 1.24 0.78 


Source: DE Usage Data 
Note: E indicates elementary schools; M indicates middle schools; H indicates high schools. 


Goal: Costs for technology per pupil for DE decline over time. 
Table G.5 shows the following trends for students: 
e The total number of students who used DE increased slightly each year, but the frequency with 
which those students used the DE resources decreased in Year 3. 
e The cost per student each year for the use of DE was about $8.00 per student with a small 
decrease across years. 
e Onthe other hand, given the decreased number of times that students accessed these 
resources (authentications), the cost per authentication increased from $1.77 in Year 1 to $2.15 
in Year 3. 


For teachers: 
e The number of teachers who utilized DE resources and their total number of authentications 
decreased each year. Because fewer teachers used the resources, the average cost per teacher 
increased from $39.13 in the first year to $64.52 in the final year. 


e Similarly, the cost per teacher authentication increased from $2.50 in the first year to $3.82 in 
the third year. 
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The total number of users (teachers and students) remained relatively consistent across years because 
more students and fewer teachers used the resources across years. However, the cost per active user 
(teacher or student) and cost per authentication for teachers and students increased across years. This 
is because the teachers and students used the resources less often in the third year, resulting in an 
increased cost per active user from $6.75 in year one to $7.12 in year three. The cost per authentication 
also increased from $1.04 in year one to $1.37 in year three. While these increases are not large, they 
are not in the desired direction, suggesting that teachers and students were not utilizing the DE 


resources with the frequency and intensity intended by the initiative. 


Table G.5 
Discovery Education Costs 


Year 1 Year 2 Year 3 


Total Students who Utilized DE 47,961 52,199 52,539 
Total Authentications (Students Only) 221,607 268,722 195,388 
Sh dents Average Authentications per Student 4.62 5.15 3.72 
License Costs $391,373 $412,020 $420,640 
Cost Per Pupil $8.16 $7.89 $8.01 
Cost Per Authentication (Students Only) $1.77 $1.53 $2.15 
Year 1 Year 2 Year 3 
Total Teachers who Utilized DE (Teachers) 10,001 7,530 6,519 
Total Authentications (Teachers Only) 156,347 142,669 110,163 
Teachers Average Authentications per Teacher 15.63 18.94 16.90 
License Costs $391,373 $412,020 $420,640 
Cost Per Active Users (Teachers) $39.13 $54.71 $64.52 
Cost Per Authentication (Students & Teachers) $2.50 $2.88 $3.82 
Year 1 Year 2 Year 3 
Total Users who Utilized DE (Students & Teachers) 57,962 59,729 59,058 
Total Authentications (Students & Teachers) 377,954 411,391 305,551 
Svarall Average Authentications per User 6.52 6.89 5.20 
License Costs $391,373 $412,020 $420,640 
Cost Per Active Users (Students & Teachers) $6.75 $6.89 $7.12 
Cost Per Authentication (Students & Teachers) $1.04 $1.00 $1.37 


Source: DE usage data and DE cost data 


Goal: Increase in the number of schools integrating technology appropriately into instruction. 

The number of schools with teachers and students utilizing the DE resources did not increase as desired 
across time. Table G.6 shows the ratio of elementary, middle, and high schools with students and 
teachers using DE resources, along with the percentage increase in the number of schools compared to 
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the year prior. During the first year, more than half of the elementary schools and nearly three-quarters 
of the middle and high schools had at least one teacher using DE resources. About half of the 
elementary, middle, and high schools had at least one elementary student utilizing DE resources. In 
some schools, teachers or students were utilizing the resources, not both. Only 34% of elementary 
schools had both teachers and students utilizing the DE resources. Half of the high schools had both 
teachers and students utilizing the resources. Unfortunately, the percentage of schools with teachers or 
students using DE resources did not increase between Years 1 and 2 or Years 2 and 3, suggesting that 
schools that used DE resources in the first year either continued or discontinued use in the second or 
third year. 


Table G.6 
Number of and Percentage Increase in Schools with Teachers and Students Using DE Resources 
Increase from Year 1 to Increase from Year 2 to Increase from Year 1 to Year 
2 3 3 
Percentage of Percentage of Percentage of 
Ratio Schools that Ratio Schools that Ratio Schools that 
Increased Increased Increased 

Teach 

e er 63/106 59.43 0/106 0 0/106 0 
Elementar 

y Students 48/106 45.28 0/106 0 0/106 0 
Both 36/106 33.96 0/106 0 0/106 0 
ma 26/36 72.22 0/36 0 0/36 0 
Middle — students 21/36 58.33 0/36 0 0/36 0 
Both 16/36 44.44 0/36 0 0/36 0 

Teach 
a of ‘99730 73.33 0/31 0 0/30 0 
High Students 16/30 53.33 0/31 0 0/30 0 
Both 15/30 50 0/31 0 0/30 0 

Teach 111/17 

ee Pe 64.53 0/173 0 0/172 0 
Overall Students 85/172 49.42 0/173 0 0/172 0 
Both 67/172 38.95 0/173 0 0/172 0 


Source: DE Usage Data 
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Appendix H 


Classroom Observations 


Methods 


Sample 

The sample for this study included 165 WCPSS schools. The four alternative schools and two Leadership 
academies were excluded (see Table H.1). A stratified random sample of 10% of the 165 schools was 
created and included schools at the elementary, middle and high school levels. The percentage of 
students who were Economically Disadvantaged (ED) at each school was also used as a control variable 
to ensure a mix of schools with varying student demographics. The method used was systematic 
random sampling using SAS Proc Surveyselect. The schools selected are shown in Table H.2. 


Table H.1 
Sample Pool of WCPSS Schools by Grade Span and Type 
Grade Range Regular Percent of Pool 
Elementary 106 64.2 
K-8 1 | 0.6 
Middle 33 20.0 
High 25 15.2 
Total 165 100.0 


Note: Four alternative schools and two leadership academies were excluded as atypical. 
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Table H.2 


TLC School Sample for Observations 2015-16 


School Name 


Percent Economically 


Disadvantaged 


307 Alston Ridge Elementary 7.19 

447 Holly Springs Elementary 16.64 
572 Underwood Magnet Elementary 22.14 
624 Willow Springs Elementary 27.31 
451 Harris Creek Elementary 33.59 
544 Rolesville Elementary 38.30 
448 Hunter Magnet Elementary 43.09 
460 Kingswood Elementary 50.86 
618 Wildwood Forest Elementary 59.72 
440 Green Elementary 71.63 
403 East Garner Elementary 80.71 
502 Mills Park Middle 8.71 
450 Holly Ridge Middle 24.51 
592 Wake Forest Middle 42.92 
408 East Millbrook Magnet Middle 69.43 
473 Leesville Road High 18.31 
412 Enloe Magnet High 32.74 


TLC 2013-14 to 2015-16 


Instrument 

DRA created an observation tool adapted from the 
International Society for Technology in Education (ISTE) 
Classroom Observation Tool (ICOT) and the Florida Center for 
Instructional Technology Integration Matrix Scale (Welsh, 
Harmes, & Winkelman, 2011). The ICOT is a free online 
resource helpful in assessing technology integration in 
classrooms and it reflects ISTE Student Standards from 2007 
(ISTE, 2009). The TIMS Matrix (as reflected in Discovery 
Education’s DDIS tool) assessed the interaction of the learning 
environment with the level of technology integration into the 
curriculum. DRA added descriptive and open-ended 
questions to gain a more comprehensive understanding of 
the actual lessons and technology’s role in them. Examples of 
each level are included in the blue sidebars showing 
increasing levels of integration. 


Data Collection 

Each school was allotted four TLC member spots. Each of the 
six observers attempted to observe all TLC members at two or 
three schools in a geographic area. Steps taken included: 


e Principals were notified of the purpose of the visit and 
to verify who the TLC members were, 

e Teachers were then emailed about the purpose of the 
observations and the week an observer would be at 
the school. Teachers were asked to note if any times 
would not work, but were not told the actual day that 
the observation would occur (to keep instruction as 
natural as possible). Middle and high schools typically 
provided times to avoid due to other commitments. 


We found various situations which impacted our sample size 
and scheduling. At some schools, the TLC team was not full (4 
members), some teachers were on leave or tracked out, some 
had student teachers (and were therefore not teaching), and 
some had committee/team obligations that made some days 
impossible for observations. Additionally, some schools had 
field trips or school picture days during this period which 
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TIMS ENTRY LEVEL* 
GRADE 5 SAMPLE 


Lesson: The goal of this lesson was to 
review the functions of the main 
human body systems. 


Teacher: During this portion of the 
lesson, the teacher showed the class a 
PowerPoint presentation she created 
with one or two slides that highlighted 
the parts of a body system and the 
function of that system in the body. On 
at least two occasions, the teacher 
played familiar songs where the lyrics 
were changed to describe a body 
system. 


Students: The students responded as 
a whole group to the teacher’s 
questions about different body 
systems. They were asked to make 
notes so they would have reference 
study materials for an upcoming test. 


Rationale: This lesson was rated as 
entry (the lowest level of integration) 
because the teacher delivered all of the 
alcelaaateiulolaMuiall(omaat=mcyablol-lalacm e)l-hVZ-1e| 
a passive role. The use of technology in 
this setting was very conventional with 
no student choice or opportunity for 
(>-de) (ol e-lulelam-lare mere) i=] ele)e-la (elem 


* The level of technology integration is one 
way to classify lessons. See TIMS Section 
later in this appendix for level descriptions. 


made scheduling more difficult. The number of situations that pulled teachers out of their classrooms 
was somewhat surprising. In a few cases, teacher commitments meant teachers knew observations 


would be one of two days. 


Overall, 49 of the 68 observations were completed (72%), as shown below. 
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Table H.3 
TLC Members by School and Observation Completion 


# # % 

Observations Observations of Sample 
Level/# Schools Expected Completed (0) oXX-1 a (10 | 
Elementary 44 32 72.7% 
n=11 
Middle n=4 16 | 9 56.3% 
High n=2 8 | 8 100.0% 
Total 68 49 72.1% 


Some of those who could not be observed were asked for an example of how they used what they 
learned in TLC and whether they found it valuable. 


Technology was used by TLC members in a wide 
How Did TLC Members Use Digital Resources? variety of ways in the observations, representing 
all levels of integration into instruction. Some 
were quite basic, such as watching videos, while 
others were more advanced, such as students 
selecting a topic and then utilizing various 
technology to find information and produce a 
presentation or report. Technology was more 


What technology was used and how? 


A wide range of technology was utilized by students and 
teachers in the observed lessons. Some of the most 


common were computers (desktop and laptop), often considered as appropriate, but not necessary 
interactive white boards, iPads, Smartphones/handhelds, for a lesson. This implies technology was used as a 
presentation software, and Google documents and way to enhance the learning. Student 


spreadsheets. Less commonly noted were video cameras, engagement was high. Teachers most often 
televisions, Website builders, response system, Discovery ™odeled discussion or provided interactive 
Education, Splash Math, Wikis, assessment software, and _ direction, but also often facilitated or supported 


graphics. Teachers also occasionally used document student learning. DRA staff observed examples of 
cameras all student ISTE standards. 


In a few of the observations, teachers used one device with the whole class; in others two or three 
students shared a device (perhaps an iPad or laptop or a combination); and all other students had a 
device to work on individually (through the use of a laptop cart, a computer lab, or as part of a Bring 
Your Own Device effort). There were instances when students also rotated through activities or centers 
which did and not involve digital learning. In addition, technology was used to display timers and 
student group membership, to create graphic organizers, and to prepare reports and presentations. 
Specific examples of lessons are included by type based on the TIMS framework in the summary of this 
report and in this appendix. 


In most of the observations, full classes of students were observed. The average number of students 
was 21 overall, with secondary observations having more students on the average than the elementary 
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observations. The number of students per device available was lower in high school and elementary 
classrooms than in middle schools. Access was usually best when students worked in a computer lab 
setting. 


Table H.4 

Number of Devices and Students Observed by Grade Span 

Students 
Average # Average # Per 

Students Devices Device 

Overall 21.0 7.2 3.4 
Elementary 19.3 5.3 2] 
Middle 23.6 15.4 6.6 
High 24.9 6.0 2.4 


Source: Classroom observations (n=49) 


Did the use of technology enhance the instruction? 
e Was it needed for the lessons? 


e Was it used appropriately? 
e What was the level of student engagement? 


As Figure H.1 shows, average ratings for the level of need for technology and the appropriateness of 
technology were both above the midpoint on the five-point scale. The level of engagement was also 
above the midpoint on the scale. 


Need for digital resources was rated somewhat lower than the appropriateness of its use (roughly 3.5 
versus 4.3 across subjects, respectively). Thus, technology use was often considered appropriate even 
when it was not necessary for a lesson. In that sense, teachers saw technology as enhancing their 
instruction. In terms of student engagement, ratings also were fairly high (4.1 in middle and 4.2 in 
elementary) except in high school, which was 3.1 (just above the scale midpoint). Middle schools had 
the highest ratings for need and appropriateness. 
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Figure H.1 
Need and Appropriateness of Technology and Student Engagement by Grade Span 
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Overall Elementary Middle High 
Level (n) Need Appropriateness Student Engagement 
Elementary (32) 3.4 4.1 4.2 
Middle (9) 3.9 4.8 41 
High (8) 3.4 AA a1 
Overall (49) 3.5 4.2 4.0 


Source: TLC Observations, Spring 2015. 


Scales: Need for Technology: Not needed (1) to essential (5) 
Appropriateness of Technology: Not appropriate (1) to very appropriate (5) 
Level of Student Engagement: Not engaged (1) to very engaged (5) 


n= number of observations 


When examined by subject (see Figure H.2), the pattern of responses was similar to those by grade 
span, with appropriateness highest, engagement next, and need lowest (but still above the midpoint of 
the scale). The need and appropriateness and engagement of students varied somewhat by subject. 
Small sample sizes should lead to cautious interpretations of the following trends: 


e Science was observed as having the highest need for technology in the lesson and the second 
highest appropriateness. 


Math was observed to have the lowest need for technology and the lowest appropriateness, but 


student engagement levels were higher when technology was used. 


e The Multiple Subjects category scored the highest for level of student engagement. Nine out of 
14 of these observations combined Language Arts with another subject. 
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Figure H.2 
Need and Appropriateness of Technology and Student Engagement by Subject 
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« Engaged 4.0 3.8 4.0 3.9 4.1 


@Need Appropriateness « Engaged 


Source: TLC Classroom Observations, Spring 2015 Scale 1-5 
n=number of observations. Total was 49, but sample sizes by subject were small. 


What were the teacher and student roles? 
Technology Integration Matrix (TIMS) 


The Technology Integration Matrix (TIMS) categorizes lessons according to the level of technology 
integration (i.e., entry, adoption, adaptation, infusion, or transformation). Within these levels, lessons 
are further characterized based on the learning environment (i.e., active, collaborative, constructive, 
authentic, and goal directed). The full scale is shown in Table H.7. An example of each level on the 
technology integration scale is shown in sidebars in this report and in Figure H.6 along with the 
characteristics of the learning environment. 


All levels of technology integration were observed, which was a positive finding. More lessons were in 
the middle of the matrix than at the extremes as shown in Table H.5. This indicates teachers had 
generally gotten past the use of technology per se (although it was sometimes found useful for basic 
tasks) and had adopted use of technology in conventional ways as well as adapting lessons to give 
students more opportunity to work independently or in collaborative groups. 
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Table H.5 
Percentage of Observations in Each Level of Technology Integration 


Adoption 35.0% 
Adaptation 30.9% 
Infusion 14.4% 
Transformation 9.5% 


N= 49 observations 


In terms of the characteristics of the learning environment, all levels (i.e., active, collaborative, 
constructive, authentic, and goal directed) were observed. By subject, average ratings of technology 
integration were generally highest for social studies and multiple subject lessons. By type, collaborative 
environments were rated lower for technology integration than the other types. 


Figure H.3 


Average Ratings of Learning Environment by Subject 
Based on TIMS Matrix 
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Active Collaborative Constructive Authentic Goal-Directed 

@ Language Arts 2.56 2 2.5 2.5 2.5 
@ Math 2.5 2.2 2.5 2.3 2.3 
@ Science 2.6 22 2.2 2.4 2.8 
@ Social Studies 3.22 3.25 2.89 3.33 3.33 
® Multiple 2.85 3.08 3.15 2.92 3.23 


Type of Learning Environment 
@Language Arts Math mScience wmSocialStudies # Multiple 


Note: The numbers on the Y axis correspond to the Levels of Technology Integration into the 
Curriculum according to the TIMS Matrix. 1= Entry, 2= Adoption, 3= Adaptation, 4= Infusion, and 
5= Transformation. Based on 45 observations with necessary data. 


81 


TLC 2013-14 to 2015-16 


Teacher Roles 
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One section of the Teacher Leader Corps (TLC) observation form asked for the role or roles of the 
teacher based on five categories as shown below. 


Table H.6 
Definitions for Teacher Roles 


Lecturing 


Teacher conveying information to the students with little or no interaction during the 
presentation. 


Interactive Direction 


Teacher presenting material while asking questions and eliciting prior knowledge or 
critical thinking. 


Facilitating/Coaching 


Teacher providing advice to learning activities undertaken by the students outside of 
teacher presentation. 


Modeling Teacher explicitly demonstrating behavior or procedures. Distinguished from lecturing 
by the emphasis on demonstration rather than explication. 

Moderating Group facilitation, where the teacher advises a dialog conducted among students. 

Discussion Distinguished from Interactive direction by the emphasis on student interaction with 
one another rather than response to the teacher. 

Other Teacher is monitoring students completing individual or group work. This could be an 

(Monitoring assessment, assignment, or independent reading. 

Students) 


Monitoring was the only response noted in the “Other” category. Observers could select more than one 
role, and often two or three were selected during the course of one lesson. Moderating discussion and 
interactive direction were most commonly observed (representing over 30% each of the responses). 
Facilitate/coach was also quite common with 24% of the responses. Lecturing, modeling, and 
monitoring students were least commonly observed. 


Figure H.4 
TLC Teachers’ Role During Lessons Observed 


= Lecturing 

a Facilitate/Coaching 

= Modeling 

= Moderating Discussion 
= Interactive Direction 


= Monitoring students 


Source: TLC Observations. 102 responses were given for 31 of the observations. 
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Observers had the option to provide additional notes on the role of the teacher. These comments were 
coded categorically, and provided more detail on teacher roles. Comments reflected that teachers’ 
roles were varied, and indicated teachers provided individual instruction and explanations, served as 
facilitators of student work, and supporting students as they worked. 


e = Individual instruction and explanation of the lessons accounted for close to one third of the 
coded data (10 responses or 32% of coded data). The following is one example of a teacher’s 
role: The teacher discussed the appropriate uses of technology and explained good strategies 
that some students were using which could be helpful to others. 


e Teachers worked as facilitators in their classrooms often (8 responses or 26% of coded data). 
For example, one observer noted that the teacher “facilitate[ed] discussion and interaction of 
sharing what they [knew] and adding facts to a list." 


e Teachers also functioned in a support role for their students (6 responses or 19% of coded data). 
For example, an observer noted that the "Teacher and two assistants support[ed] groups of 
students in drafting story problems and coming up with a plan. [The] teacher showed examples 
to the students of what they would create (video of his/her students acting out story problem 
last year). [The] teacher support[ed] students in working toward creating their own videos." 


Student Roles 


Small group and individual work were observed somewhat more often than work with the whole class 
or in pairs, but each student group size represented at least 20% of the responses. 


Figure H.5 
Percentage of Responses Related to Student Group Size during Observations 


35% 


30% 28.4% 29.5% 


25% - 
20.5% 21.6% 


Individuals Pairs Small Groups Whole Class 


N = 88 responses. 
Note: Multiple responses were possible within one lesson. 
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Observers indicated the occurance of a variety of activities during each observation. Trends in the types 
of activities observed were that: 


A wide variety of activities were observed, sometimes within the same lesson. 


The most frequent activities observed (11% to 14% of the time), included information analysis, 
watching a video, student discussion, drill and practice, and writing. 


Creating a presentation, giving a presentation, hands-on skills, and conducting research were 
observed 6% to 9% of the time. 


Activities observed less than 5% of the time were receiving a presentation, taking a test, or 
playing an instructional game. 


Observer notes provided more detail on the choices students were provided within lessons for 39 
observations. These qualitative data were coded into descriptive categories and indicate the following 
trends related to student choice and instructional technology available. Comments were mixed on 
student engagement. 


The most common coding category (10 of 39 responses or 26% of coded data) suggested that 
students had a choice either of the type of technology used or the particular software being 
used on a device. For example, one observer noted, "This lesson provided the students the 
opportunity to engage in their own research on a topic of their choosing. Students were able to 
select the type of device they were most comfortable using and were apparently told to bring a 
device from home prior to class." Conversely, four responses or 10% of the coded data reflected 
that the students lacked choice in the lesson observed. For example, an observer noted that 
there was "very little student choice and not a lot of critical thinking or inquiry” in that particular 
lesson. 


The second largest coding category (8 responses or 21% of coded data) indicated that students 
had access to multiple types of technologies in the lessons observed. As one observer noted, 
“{The] teacher utilized multiple technology programs and hardware seamlessly. [The teacher] 
utilized Google Documents and Edmodo in ways that built collaboration and kept parents aware 
and involved." 


ISTE Standards 


The ISTE Standards for student use of technology (2007 version) relate to creativity, communication, 
information fluency, critical thinking, digital citizenship, and technology operations. Each area has four 
standards associated with it. The full list of standards is included at the end of this appendix. As 
mentioned earlier, the largest number of observations was at the elementary level, so this list is most 
reflective of elementary TLC teachers. Activities were observed which related to all of the standards. 
These standards included: 
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e § Creativity--Apply existing knowledge to generate new ideas, products, or processes. 

e Communications-- Interact, collaborate, publish with peers in digital environments and 
communicate information ideas to audiences using a variety of media and formats were 
mentioned an equal number of times. 

e Information fluency--Locate, organize, analyze, evaluate, synthesize information or data. 

e = Critical thinking-- Plan and manage activities to develop a solution or complete a project. 

e Digital citizenship--Practice safe and responsible use of technology, and exhibit positive 
attitude. toward technology that supports collaboration were mentioned most frequently. 

e Technology operations-- Demonstrated understanding of technology being used. 


The full list of frequencies for each standard, along with the complete list of standards, is shown in Table 
H.8. Based on the 52 observations recorded, the standard most often observed was under technology 
operations (43 times). The next most common related to digital citizenship, in which two standards 
were commonly observed including practice safe and responsible use of technology and exhibit positive 
attitude toward technology that supports collaboration (35 and 34 times, respectively). These reflect 
basic skills and attitudes students must have to take advantage of technology in learning. 


The standard least often observed was under the creativity standard (5 times in 52 observations). 
Additionally, critical thinking was another standard that was seldom observed (12 times). 


Some examples of basic and advanced activities under creativity included students: 


e Completing skill and drill practice problems 

e Exploring how bee-keeping equipment works 

e Looking up characters, places, words from their books that were historical fiction focused on 
the Civil War 

e Using many different strategies to solve complex word problems 

e Creating a newsletter in Publisher or Word on one of the three regions in NC designed to 
share why people should visit or move to the area. They reviewed articles and a website to 
find information for the newsletter, and produced the newsletter in one of two software 
programs (Grade 4) 
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Characteristics of the Learning Environment 


Information 
passively 
received 


Collaborative 


Constructive Information 
delivered to 
students 


Authentic 


Goal-Directed 


Table H.7 


Levels of Technology Integration into the Curriculum 


Conventional, Choice of tools 

procedural use | i and regular, 

of tools = self-directed 
use 


Guided, Choice and 


conventional i regular use for 


use for building 
knowledge 


Conventional 
and procedural 
use of tools to 
plan or monitor 
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Transformation 


Extensive and 
unconventional 
use of tools 


Extensive and 


unconventional 
use of 
technology 
tools to build 


knowledge 


Extensive and 


higher order use 
of tools to plan 
and monitor 
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Sample Classroom Observations of TLC Members 


TIMS ADAPTATION LEVEL 
GRADE 2 SAMPLE 


Lesson: The goal of this science lesson was to 
develop student understanding of the lifecycle of a 
butterfly. 


Teacher: After the teacher showed the class a 
video about the life cycle of butterflies, she 
facilitated an interactive activity where students 
shared new information that they learned from the 
video. 


Students: Students watched a video on the SMART 
Board about the life cycle of a butterfly and then 
wrote a novel fact on a piece of paper. Sitting ina 
circle, students then crumbled their paper and had 
a “snowball fight,” which involved throwing their 
paper in the middle of the circle. Students selected 
a paper from the pile, read the fact, and added 
another novel fact. This repeated three times until 
students had a sheet of four facts. Each student 
then shared something they learned during the 
snowball fight with the class before lining up for 
lunch. 


Integration of technology was at the adaptation 
level in that the video provided the content for the 
lesson and students were allowed choice in their 
contribution to the list of facts. Furthermore, 
students had to explore and analyze previous 
contributions to identify new pieces of information 
which they could contribute to the list of facts. This 
activity stimulated collaborative accumulated 
knowledge in a virtual environment. 
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TIMS ADOPTION LEVEL 
GRADE 3 SAMPLE 


Lesson: The goal of this math level 
was to review student skills with 
fractions. The teacher led the 
students through a fractions game 
on an interactive white board. 


Grouping: Students either paired 
up and shared an iPad or worked 
independently. 


Teacher: Started with an 
interactive review of how the game 
worked, and then functioned as a 
facilitator as she led students 
through the game. Technology 
integration was primarily at the 
adoption level; students were given 
a structure to follow for the game, 
elalomag=tel0l=laldhvmere) | t=] ole) ¢-] x10 Ke) 
solve problems, but ina 
conventional manner without 
individual choices. 


Students: The 17 students 
practiced skills previously learned 
about fractions. To be successful at 
the game, students had to use prior 
knowledge to solve questions 

El ofelU] miag-leidlo)atcw-larema-vere)aemaal-iie 
responses via iPad. Students were 
allowed to create team names that 
protected their identities when 
scores were reported. Students 
were at ease and comfortable using 
technology. 
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Figure H.7 
Student ISTE Standards 


A 


iS 


Standards 


ISTE Standards 
Students 


International Society for 
Technology in Education 


1. Creativity and innovation 


Students demonstrate creative thinking, construct 
knowledge, and develop innovative products and 
processes using technology. 


b. 


Apply existing knowledge to generate new 
ideas, products, or processes 


Create original works as a means of personal 
or group expression 


. Use models and simulations to explore complex 


systems and issues 


. Identify trends and forecast possibilities 


2. Communication and collaboration 


Students use digital media and environments to 
communicate and work collaboratively, including 
at a distance, to support individual learning and 
contribute to the learning of others. 


Interact. collaborate, and publish with peers, 
experts, or others employing a variety of digital 
environments and media 


. Communicate information and ideas effectively 


to mukiple audiences using a variety of media 
and formats 


. Develop cultural understanding and global 


awareness by engaging with learners of 
other cultures 


. Contribute to project teams to produce original 


works or solve problems 
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3. Research and information fluency 


Students apply digital tools to gather. evaluate, 

and use information. 

a. Plan strategies to guide inquiry 

b. Locate, organize, analyze, evaluate, synthesize, 
and ethically use information from a variety of 
sources and media 

¢. Evaluate and select information sources and 
digital tools based on the appropriateness to 
specific tacks 


d. Process data and report results 


4. Critical thinking, problem solving, 
and decision making 


Students use critical thinking skills to plan 

and conduct research, manage projects, solve 

problems, and make informed decisions using 

appropriate digital tools and resources. 

a. Identify and define authentic problems and 
significant questions for investigation 

b. Plan and manage activities to develop a solution 
or complete a project 

¢. Collect and analyze data to identify solutions 
and/or make informed decisions 


d. Use mukiple processes and diverse 
perspectives to explore alternative solutions 
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(continued from previous page) 


5. Digital citizenship 3 6. Technology operations and concepts 


Students understand human, cukural, and societal : Students demonstrate a sound understanding 
issues related to technology and practice legaland of technology concepts, systems, and operations. 


ethical behavior. a. Understand and use technology systems 


a. Advocate and practice safe, legal, and b 


é . Select and use applications effectively 
responsible use of information and technology 


and productively 
b. Exhibit a positive attitude toward using 


technology that supports collaboration, : : 
learning, and productivity d. Transfer current knowledge to learning 


of new technologies 


c. Troubleshoot systems and applications 


c. Demonstrate personal responsibility for 
lifelong learning 

d. Exhibit leadership for digital citizenship 
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Table H.8 
Frequency of Observation of Each Student ISTE Standard 


1a) 27 2a) 24 3a) 15 4a) 17 5a) 35 6a) 43 
1b) 18 2b) 24 3b) 25 Ab) 26 5b) 34 6b) 27 
1c) 17 2c) 17 3c) 18 Ac) 18 5c) 22 6c) 14 
1d) 5 2d) 18 3d) 17 Ad) 12 5d) 23 6d) 13 


Note: 52 observations of full lessons occurred. Often more than one standard was addressed. 
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